Re: Famous operational issues

[bzs]

 > On Tue, 16 Feb 2021, John Kristoff wrote:
 > 
 > > Friends,
 > >
 > > I'd like to start a thread about the most famous and widespread Internet
 > > operational issues, outages or implementation incompatibilities you
 > > have seen.
 > >

When Boston University joined the internet proper ca 1984 I was in
charge of that group.

We accidentally* submitted an initial HOSTS.TXT file which included
some internally used one-character host names (A, B, C) and one which
began with a digit (3B, an AT 3B5), both illegal for HOSTS.TXT back
then.

This put the BSD Unix program which converted from HOSTS.TXT to Unix'
/etc/hosts format into an infinite loop filling /tmp which in those
days crashed Unix and it often couldn't reboot successfully without
manual intervention.

On many, many hosts across the internet.

I hesitate to guess a number since scale has changed so much but some
of the more heated email claimed it brought down at least half the
internet by some count.

It was worsened by the fact that many hosts pulled and processed a new
HOSTS.TXT file via cron (time-based job scheduler) at midnight so no
one was around to fix and reboot systems.

The thread on the TCP-IP mailing list was: BU JOINS THE INTERNET!

It was a little embarrassing.

Today it probably would have landed me in Gitmo.

* There were two versions, the one we used internally, and the one to
be submitted which removed those host names. The wrong one got
submitted.

-- 
-Barry Shein

Software Tool & Die| b...@theworld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
The World: Since 1989  | A Public Information Utility | *oo*

Re: Texas internet connectivity declining due to blackouts

[Karl Auer]

On Wed, 2021-02-17 at 00:27 -0500, Peter Beckman wrote:
>  Buried lines makes sense where it makes sense.

Aesthetically, burying lines always makes sense. Sadly not enough
communities (and definitely too few governments) place any value on
aesthetics at all.

I've never heard anyone, ever, say "Golly, what a lovely profusion of
cables this place has! How charmingly they subdivide the view of the
sky! How delightfully they criss-cross every street and laneway!"

I wrote this back in 2013. If I've missed anything, feel free to let me
know off-list:

https://biplane.com.au/blog/?p=276

Regards, K.

-- 
~~~
Karl Auer (ka...@biplane.com.au)
http://www.biplane.com.au/kauer

GPG fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170
Old fingerprint: 8D08 9CAA 649A AFEF E862 062A 2E97 42D4 A2A0 616D

Re: Texas internet connectivity declining due to blackouts

[Peter Beckman]

On Tue, 16 Feb 2021, Rod Beck wrote:


Are the power lines buried like in Europe where I live?

I really think using poles is crazy and global warming guarantees enough
atmospheric turbulence to make it untenable. Florida is moving to bury
power lines.


Only 41% of European lines are underground [1]. Population density is
higher in the UK, 280 per sq km, versus the US, 34 per sq km [2].

Netherlands: 423 per sq km
Belgium: 376 per sq km
Germany: 233 per sq km
Switzerland: 208 per sq km
Italy: 200 per sq km

When population density is low, the cost to install buried lines does
not make financial sense, even considering the outages.

In major cities, lines are buried in the US.

Granted, there are several US States that individually are similar to
Europe:

New Jersey: 467 per sq km
Massachussetts: 331 per sq km
New York: 161 per sq km (despite having NYC, largest city in the US)
California: 95 per sq km (despite having LA, 2nd largest city in the US)
Texas: 39 per sq km

Buried lines makes sense where it makes sense. Comparing Europe to the
US is way too broad, and I don't know where you live.


[1] 
https://www.bloomberg.com/news/articles/2019-03-05/why-europe-pays-less-than-u-s-to-put-power-lines-underground
[2] 
https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population_density



From: NANOG  on behalf of 
Mikael Abrahamsson via NANOG 
Sent: Tuesday, February 16, 2021 9:06 AM
To: Sean Donelan 
Cc: nanog@nanog.org 
Subject: RE: Texas internet connectivity declining due to blackouts

On Mon, 15 Feb 2021, Sean Donelan wrote:


Strange the massive shortages and failures are only in one state.

The extreme cold weather extends northwards across many states, which aren't
reporting rolling blackouts.


https://www.texastribune.org/2011/02/08/texplainer-why-does-texas-have-its-own-power-grid/

Going at it alone can be beneficial sometimes, sometimes it's not.

--
Mikael Abrahamssonemail: swm...@swm.pp.se



---
Peter Beckman  Internet Guy
beck...@angryox.com http://www.angryox.com/
---

RE: Texas internet connectivity declining due to blackouts

[Peter Beckman]

On Tue, 16 Feb 2021, Robert Jacobs wrote:


How about letting us Texans have more natural gas power plants or even
let the gas be delivered to the plants we have so they can provide more
power in an emergency.  Did not help that 20% of our power is now wind
which of course in an ice storm like we are having is shut off... Lots of
issues and plenty of politics involved here..


 Turns out that you Texans already get a majority of your power from
 Natural Gas.

 So there's already a significant amount of power from natural gas already.

 Things I learned about the most-of-Texas Grid today:

- Natural Gas plants provide MORE THAN HALF of their total electricity
generation in 2019 (WOW!)
- Texas has their own grid to avoid Federal regulation.
- Texas does have some links to other grids but they don't trigger 
federal
regulation for some reason.
- Texas is the largest energy-producing and energy-consuming state in
the nation. The industrial sector, including its refineries and
petrochemical plants, accounts for half of the energy consumed in the
state.
- 5 Gigawatts of coal-fired capacity has retired since 2016, and
supplies 20% of power currently.
- Wind power provided about 17% of their usage
- There are two nuclear plants in Texas, only providing 10% of power.
- One of those nuclear plants are offline due to weather-related issues.

From the WashPost: 

"The Texas grid got crushed because its operators didn’t see the need to
prepare for cold weather"
https://www.washingtonpost.com/business/2021/02/16/ercot-texas-electric-grid-failure/

---
Peter Beckman  Internet Guy
beck...@angryox.com http://www.angryox.com/
---

Re: [EXTERNAL] Re: dumb question: are any of the RIR's out of IPv4 addresses?

[Mann, Jason via NANOG]

Any recommendations for legitimate ip brokers?


From: NANOG  on behalf of Michael Thomas 

Sent: Tuesday, February 16, 2021 5:46 PM
To: Fred Baker 
Cc: nanog@nanog.org 
Subject: [EXTERNAL] Re: dumb question: are any of the RIR's out of IPv4 
addresses?


On 2/16/21 4:18 PM, Fred Baker wrote:
> You may find this article interesting:
> https://urldefense.com/v3/__https://blog.apnic.net/2019/12/13/keep-calm-and-carry-on-the-status-of-ipv4-address-allocation/__;!!GaaboA!999i8DMj5mceMG2R6J8wgZ29XjBhQvAJU3QMixqhvjqpQCsdAvcck6BpWKVqMw$
>   >
>
So aside from Afrinic, this is all being done on the gray market?
Wouldn't you expect that price to follow something like an exponential
curve as available addresses become more and more scarce and unavailable
for essentially any price?

Mike


> Sent from my iPad
>
>> On Feb 16, 2021, at 3:07 PM, Michael Thomas  wrote:
>>
>> 
>> Basically are there places that you can't get allocations? If so,
>> what is happening?
>>
>> Mike
>>

Re: Texas internet connectivity declining due to blackouts

[Kevin East]

100%.  Our system has been on stage 2 aux heat (electric) ever since we
dropped below 24 or so.  Usually we might see it for a few hours on the
coldest nights.  I'd say most people are probably pulling full summer load
+20%.

On Tue, Feb 16, 2021, 5:10 PM Seth Mattinen  wrote:

> On 2/16/21 09:49, Michael Thomas wrote:
> >
> > On 2/16/21 8:50 AM, John Von Essen wrote:
> >> I just assumed most people in Texas have heat pumps- AC in the summer
> >> and minimal heating in the winter when needed. When the entire state
> >> gets a deep freeze, everybody is running those heat pumps non-stop,
> >> and the generation capacity simply wasn’t there. i.e. coal or natural
> >> gas plants have some turbines offline, etc.,. in the winter because
> >> historically power use is much much less. The odd thing is its been
> >> days now, those plants should be able to ramp back up to capacity -
> >> but clearly they haven’t. Blaming this on wind turbines is BS. In
> >> fact, if it weren’t for so many people in Texas with grid-tie solar
> >> systems, the situation would be even worse.
> >
> > You'd think that mid-summer Texas chews a lot more peak capacity than
> > the middle of winter. Plus I would think a lot of Texas uses natural gas
> > for heat rather than electricity further mitigating its effect on the
> grid.
> >
>
> The difference is that in extreme cold heat pump systems are likely
> switching on emergency heat (i.e. plain old resistance heaters) when the
> compressor alone can no longer keep up with call for heat demand, which
> requires significantly more power. That's never happening in the summer,
> which is only ever running the compressor.
>

Re: Texas internet connectivity declining due to blackouts

[Marco Belmonte]

Can you let us know how you access the information you are seeing for 
Texas? I went to the website and can't find anything that allows me to 
actually view some data other than a twitter feed.


On 2/15/2021 5:53 PM, Sean Donelan wrote:
Not as bad as Myanmar (14%), Internet connectivity in Texas has been 
declining today.  According to NetBlocks, which normally monitors 
government imposed outages, reports network connectivity at 68% in Texas.


https://netblocks.org/

Texas operates a separate electric grid, with limited interconnections 
to the rest of North America.  For political reasons


For those with long memories, ENRON a Texas based corporation, once 
upon a time drove rolling blackouts across California in order to make 
billions.

Re: dumb question: are any of the RIR's out of IPv4 addresses?

[William Herrin]

On Tue, Feb 16, 2021 at 4:47 PM Michael Thomas  wrote:
> So aside from Afrinic, this is all being done on the gray market?

Hi Mike,

No. The market is fully above board with policies written into the RIR
operations to intentionally support its existence. In the ARIN region
that's things like the "specified transfer" policy.

> Wouldn't you expect that price to follow something like an exponential
> curve as available addresses become more and more scarce and unavailable
> for essentially any price?

Most likely, but it's a long curve and we're very, very early on it.
Still lots of eyeballs using public IPv4 addresses who wouldn't be
significantly inconvenienced if they had to share an IPv4 address with
their neighbors. Eventually we'll reach a sharper part of the curve
and suddenly: IPv6.

Regards,
Bill Herrin

On Tue, Feb 16, 2021 at 4:47 PM Michael Thomas  wrote:
>
>
> On 2/16/21 4:18 PM, Fred Baker wrote:
> > You may find this article interesting:
> > https://blog.apnic.net/2019/12/13/keep-calm-and-carry-on-the-status-of-ipv4-address-allocation/
> > 
> >
> So aside from Afrinic, this is all being done on the gray market?
> Wouldn't you expect that price to follow something like an exponential
> curve as available addresses become more and more scarce and unavailable
> for essentially any price?
>
> Mike
>
>
> > Sent from my iPad
> >
> >> On Feb 16, 2021, at 3:07 PM, Michael Thomas  wrote:
> >>
> >> 
> >> Basically are there places that you can't get allocations? If so,
> >> what is happening?
> >>
> >> Mike
> >>



-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: dumb question: are any of the RIR's out of IPv4 addresses?

[Eric Kuhnke]

That depends on your definition of grey market, there is an officially
approved ARIN IP block transfer process for people who are buying, via
brokers, discrete /24s and larger.



On Tue, Feb 16, 2021, 4:46 PM Michael Thomas  wrote:

>
> On 2/16/21 4:18 PM, Fred Baker wrote:
> > You may find this article interesting:
> >
> https://blog.apnic.net/2019/12/13/keep-calm-and-carry-on-the-status-of-ipv4-address-allocation/
> > <
> https://blog.apnic.net/2019/12/13/keep-calm-and-carry-on-the-status-of-ipv4-address-allocation/
> >
> >
> So aside from Afrinic, this is all being done on the gray market?
> Wouldn't you expect that price to follow something like an exponential
> curve as available addresses become more and more scarce and unavailable
> for essentially any price?
>
> Mike
>
>
> > Sent from my iPad
> >
> >> On Feb 16, 2021, at 3:07 PM, Michael Thomas  wrote:
> >>
> >> 
> >> Basically are there places that you can't get allocations? If so,
> >> what is happening?
> >>
> >> Mike
> >>
>

Re: Starlink

[A. Pishdadi]

Did anyone from starlink contact you? I would like someone to contact me
also.

On Mon, Feb 8, 2021 at 6:45 PM Robert DeVita  wrote:

> Can someone from Starlink please contact me off list?
>
>
>
> Thank you
>
>
>
> Rob
>
>
>
>
>
> [image: photo]
>
> Robert DeVita
> CEO & Founder
>
> 
>
> 
>
> 469-581-2160
>
>  469-441-8864
>
>  radev...@mejeticks.com
>
>  www.mejeticks.com 
>
>   3100 Carlisle St
> ,
> 16-113, Dallas TX 75204
>
>
>
>
>
-- 
Thanks,
Ameen Pishdadi
Gigenet.com  / am...@gigenet.com

Re: Famous operational issues

[Rich Kulawiec]

On Tue, Feb 16, 2021 at 01:37:35PM -0600, John Kristoff wrote:
> Which examples would make up your top three?

Morris worm, November 1988.  Much confusion and eventually the realization
the John Brunner had called it from 13 years out ("The Shockwave Rider", 1975).
But sloppy coding meant it could be defeated with one line of /bin/sh.

---rsk

Re: Famous operational issues

[Richard Golodner]

That was the one with the most severe imact for my company. Seven Frame 
Circuits (UUNET) and we all saw what an updtae can do


On 2/16/21 3:28 PM, Sean Donelan wrote:

Since you said operational issues, instead of just outage...

How about MCI Worldcom's 10-day operational disaster in 1999.


http://www.cnn.com/TECH/computing/9908/23/network.nono.idg/
How not to handle a network outage

[...]
MCI WorldCom issued an alert to its sales force, which was given the 
option to deliver a notice to customers by e-mail, hand delivery or 
telephone – or not at all. After a deafening silence from company 
executives on the 10-day network outage, MCI WorldCom CEO Bernie 
Ebbers finally took the podium to discuss the situation. How did he 
explain the failure, and reassure customers that the network would not 
suffer such a failure in the future? He didn't. Instead, he blamed 
Lucent.

[...]

Re: dumb question: are any of the RIR's out of IPv4 addresses?

[Michael Thomas]

On 2/16/21 4:18 PM, Fred Baker wrote:
You may find this article interesting: 
https://blog.apnic.net/2019/12/13/keep-calm-and-carry-on-the-status-of-ipv4-address-allocation/ 



So aside from Afrinic, this is all being done on the gray market? 
Wouldn't you expect that price to follow something like an exponential 
curve as available addresses become more and more scarce and unavailable 
for essentially any price?


Mike



Sent from my iPad


On Feb 16, 2021, at 3:07 PM, Michael Thomas  wrote:


Basically are there places that you can't get allocations? If so, 
what is happening?


Mike

Re: Texas internet connectivity declining due to blackouts

[Haudy Kazemi via NANOG]

On Tue, Feb 16, 2021, 17:12 Seth Mattinen  wrote:

> On 2/16/21 09:49, Michael Thomas wrote:
> >
> > On 2/16/21 8:50 AM, John Von Essen wrote:
> >> I just assumed most people in Texas have heat pumps- AC in the summer
> >> and minimal heating in the winter when needed. When the entire state
> >> gets a deep freeze, everybody is running those heat pumps non-stop,
> >> and the generation capacity simply wasn’t there. i.e. coal or natural
> >> gas plants have some turbines offline, etc.,. in the winter because
> >> historically power use is much much less. The odd thing is its been
> >> days now, those plants should be able to ramp back up to capacity -
> >> but clearly they haven’t. Blaming this on wind turbines is BS. In
> >> fact, if it weren’t for so many people in Texas with grid-tie solar
> >> systems, the situation would be even worse.
> >
> > You'd think that mid-summer Texas chews a lot more peak capacity than
> > the middle of winter. Plus I would think a lot of Texas uses natural gas
> > for heat rather than electricity further mitigating its effect on the
> grid.
> >
>
> The difference is that in extreme cold heat pump systems are likely
> switching on emergency heat (i.e. plain old resistance heaters) when the
> compressor alone can no longer keep up with call for heat demand, which
> requires significantly more power. That's never happening in the summer,
> which is only ever running the compressor.
>


Modern air source heat pumps, including air to water units, do not need to
fallback to resistance until somewhere in the -4 to -22 degrees F range,
depending on ASHP model. That is colder than the lowest lows reported so
far in TX during the current polar vortex. Older units from say 30 years
ago had significantly higher cutover points. I'm guessing the installed
equipment base in TX probably includes a lot of older units.

The difference is while old air source heat pumps were enough to provide
all the HVAC needs in moderate temps, modern units can also provide all the
heating needs in cold climates like found in Minnesota and Wisconsin, all
while maintaining a COP > 1.0, i.e. better than resistance.

Building energy performance also matters. Leaky buildings can expect high
energy requirements as the desired interior temperature diverges from the
exterior temperature. Well built homes can be heated on nothing more than
the output of a regular toaster.

I read that part of the TX issue was a natural gas supply shortfall, where
natgas was prioritized to heating applications, leaving electric power
generation short. MicroCHP and/or district heating tied into available heat
sources (maybe also to datacenter cooling?) would be of great benefit in
keeping the lights on and places warm.

The attempts to place blame on renewables are disingenuous distractions
away from infrastructural design weaknesses that are being exposed by
stressed systems. There are examples of renewables working fine, in colder
regions, with high (up to 100%) fractions of energy coming from renewable
sources. These systems tend to maximize the use of every available BTU or
kWh, and they don't try to solve everything by just throwing more BTUs and
kWh at the problem. For starters, there is a relatively simple geothermal
system, designed by a man in Nebraska, that allows him to grow citrus:

https://www.youtube.com/watch?v=ZD_3_gsgsnk

https://greenhouseinthesnow.com

>

Re: Texas internet connectivity declining due to blackouts

[Fred Baker]

True, Sean, but Texas has its own ISO. The counterpart wouldn’t be “Delaware 
has rolling blackouts”, but “The Eastern ISO has following blackouts”.

Sent from my iPad

> On Feb 15, 2021, at 8:49 PM, Sean Donelan  wrote:
> 
> 
> 
>> On Tue, 16 Feb 2021, Cory Sell via NANOG wrote:
>> adoption. Sure, wind isn’t perfect, but looks like solution relied on failed
>> in a massive way.
> 
> Strange the massive shortages and failures are only in one state.
> 
> The extreme cold weather extends northwards across many states, which aren't 
> reporting rolling blackouts.

Re: [EXTERNAL] Re: dumb question: are any of the RIR's out of IPv4 addresses?

[Elvis Daniel Velea]

Hi,

> On Feb 16, 2021, at 15:42, Mann, Jason via NANOG  wrote:
> 
> 
> Are their legtimate websites to go to purchase new blocks?

look at https://www.v4escrow.com as it may be what you are looking for.

Elvis

> 
> From: NANOG  on behalf of Jennifer 
> Sims 
> Sent: Tuesday, February 16, 2021 4:10 PM
> To: Michael Thomas 
> Cc: nanog@nanog.org 
> Subject: [EXTERNAL] Re: dumb question: are any of the RIR's out of IPv4 
> addresses?
>  
> Pretty sure APNIC is out of addresses.
> 
> It's now a matter of "when someone sells a block, you buy a block".
> 
> On Wed, Feb 17, 2021 at 10:07 AM Michael Thomas  wrote:
> 
> Basically are there places that you can't get allocations? If so, what 
> is happening?
> 
> Mike
> 

Re: dumb question: are any of the RIR's out of IPv4 addresses?

[Fred Baker]

You may find this article interesting: 
https://blog.apnic.net/2019/12/13/keep-calm-and-carry-on-the-status-of-ipv4-address-allocation/

Sent from my iPad

> On Feb 16, 2021, at 3:07 PM, Michael Thomas  wrote:
> 
> 
> Basically are there places that you can't get allocations? If so, what is 
> happening?
> 
> Mike
> 

Re: [EXTERNAL] Re: dumb question: are any of the RIR's out of IPv4 addresses?

[Niels Bakker]

* nanog@nanog.org (Mann, Jason via NANOG) [Wed 17 Feb 2021, 00:44 CET]:

Are their legtimate websites to go to purchase new blocks?


IPv4 is not like Bitcoin, new addresses aren't being mined using 
gigantic amounts of electricity at enormous environmental cost.



-- Niels.

Re: Texas internet connectivity declining due to blackouts

[Brett Frankenberger]

On Tue, Feb 16, 2021 at 08:02:38AM +0200, Mark Tinka wrote:
> 
> On 2/16/21 07:49, Matthew Petach wrote:
> 
> > Isn't that a result of ERCOT stubbornly refusing to interconnect with
> > the rest of the national grid, out of an irrational fear of coming under
> > federal regulation?
> > 
> > I suspect that trying to be self-sufficient works most of the time--but
> > when you get to the edges of the bell curve locally, your ability to be
> > resilient and survive depends heavily upon your ability to be supported
> > by others around you.  This certainly holds true for individual humans;
> > I suspect power grids aren't that different.
> 
> If there was a state-wide blackout, they'd need to restart from the national
> grid anyway. 

The Texas Grid has black-start capability.  In the event of a
state-wide blackout, they would not restart from the Eastern or Western
US Grid.

> Why not have some standing interconnection agreement with them
> anyway, that gets activated in cases such as these?

They have 820MW of interconnection with the Eastern Interconnect (the
Eastern US grid).  During most of this, it's been moving nearly 820MW
into Texas.  (Three were power shortages and rolling blackouts in
portions of the Eastern Interconnect also, although for much shorter
windows of time.  During those times, less power was flowing into
Texas, presumably because the Eastern Interconnect didn't have it
available (in the right places).)

Connections are more expensive that just a transmission line, because
you have to go AC-DC-AC (or have a rotary frequency converter).

> Sorry, unfamiliar with U.S. politics in this regard, so just doing 1+1.

Three grids, Western, Eastern, and Texas.  A GW or so of DC ties
between the Eastern and Western; nothing between the Western and Texas
(directly), and, as noted above, 880MW between the Eastern and Texas. 
(Very roughly, that's 2% of peak demand for the Texas grid.)

Eastern and Western exist largely for technical reasons (too big to
keep synchronized, at least without building a lot more ties between
them).  Texas is independent largely for political reasons.

 -- Brett

Re: Famous operational issues

[Mark Andrews]

> On 17 Feb 2021, at 09:51, Sean Donelan  wrote:
> 
> 
> Biggest internet operational SUCCESS
> 
> 1. Secure Shell (SSH) replaced TELNET. Nearly eliminated an entire class of 
> security problems on the Internet.  But then HTTP took over everything, so a 
> good news/bad news.
> 
> 2. Internet worms massively reduced by changed default configurations and 
> default firewalls (Windows XP proved defaults could be changed). Still need 
> to work on DDOS amplification.
> 
> 3. Head of Line blocking in IX switches (although I miss Stephen Stuart 
> saying "I'm Sorry" at every NANOG for a decade). Was a huge problem, which is 
> a non-problem now.
> 
> 4. Classless Inter-Domain Routing and BGP4 changed how Internet routing 
> worked across the entire backbone, and it worked!  Vince Fuller et al rebuilt 
> the aircraft in flight, without crashing.
> 
> 5. Y2K was a huge suggess because a lot of people fixed things ahead time, 
> and almost nothing crashed (other than the National Security Agency's 
> internal systems :-).  I'll be retired before Y2038, so that's someone else's 
> problem.

Lets hope you aren’t depending on a piece of medical equipment with a Y2038 
issue to keep you alive.

Y2038 is everybody's problem!

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742  INTERNET: ma...@isc.org

Re: dumb question: are any of the RIR's out of IPv4 addresses?

[Rubens Kuhl]

On Tue, Feb 16, 2021 at 8:06 PM Michael Thomas  wrote:

>
> Basically are there places that you can't get allocations? If so, what
> is happening?
>
>
In LAC region (LACNIC, NIC.br and NIC.mx), the controlled depletion phases
are now complete and the RIR reached 0 available IPv4 addresses, regardless
of request type (new entrants or not, small or large allocations). There is
a continuous reclaim process that from time to time reallocates previously
allocated addresses, or blocks that might come from IANA reclaim process.
For the requests made in 2020(only new entrants up to /22 allowed), odds
are all got or will get their block this year. For the ones requesting this
year, the outlook is not so favorable and it might take years. Or possibly
never.


Rubens

Re: [EXTERNAL] Re: dumb question: are any of the RIR's out of IPv4 addresses?

[Mann, Jason via NANOG]

Are their legtimate websites to go to purchase new blocks?


From: NANOG  on behalf of Jennifer Sims 

Sent: Tuesday, February 16, 2021 4:10 PM
To: Michael Thomas 
Cc: nanog@nanog.org 
Subject: [EXTERNAL] Re: dumb question: are any of the RIR's out of IPv4 
addresses?

Pretty sure APNIC is out of addresses.

It's now a matter of "when someone sells a block, you buy a block".

On Wed, Feb 17, 2021 at 10:07 AM Michael Thomas 
mailto:m...@mtcc.com>> wrote:

Basically are there places that you can't get allocations? If so, what
is happening?

Mike

Re: Famous operational issues

[Joe]

If were just talking about outages historically, I recall the 1996 AOL
Email debacle, not really anything to do with network mishaps but more so
DNS configuration..

As well, I believe the North East 2003 blackout was a great DR test that no
one was expecting.

Of course we also have the big non-events too such as Y2K

Regards
-Joe B.


On Tue, Feb 16, 2021 at 1:38 PM John Kristoff  wrote:

> Friends,
>
> I'd like to start a thread about the most famous and widespread Internet
> operational issues, outages or implementation incompatibilities you
> have seen.
>
> Which examples would make up your top three?
>
> To get things started, I'd suggest the AS 7007 event is perhaps  the
> most notorious and likely to top many lists including mine.  So if
> that is one for you I'm asking for just two more.
>
> I'm particularly interested in this as the first step in developing a
> future NANOG session.  I'd be particularly interested in any issues
> that also identify key individuals that might still be around and
> interested in participating in a retrospective.  I already have someone
> that is willing to talk about AS 7007, which shouldn't be hard to guess
> who.
>
> Thanks in advance for your suggestions,
>
> John
>

Re: Texas internet connectivity declining due to blackouts

[Michael Thomas]

On 2/16/21 3:19 PM, Sabri Berisha wrote:

- On Feb 16, 2021, at 6:28 AM, Michael Thomas m...@mtcc.com wrote:


We use propane. It's less dense energy-wise than gasoline, but it's
really easy to switch over.

Why not use both? Plenty of generators that are dual fuel out there.
Last year I converted my Duramax to dual fuel by replacing the
carburator. Easy-peasy.


gasoline has a shelf life, though with PG that isn't a problem :/

but the larger issue is that i really would prefer not have a bunch of 
gasoline around. it's messier too in comparison to just switching a 
propane tank. we have like three or four 5 gallon tanks which we use in 
the mean time for bbq's, etc. we manage to run the things we need for 
about 24 hours on one tank.


Mike

Re: dumb question: are any of the RIR's out of IPv4 addresses?

[George Michaelson]

On Wed, Feb 17, 2021 at 9:21 AM Christopher Morrow
 wrote:
>
> On Tue, Feb 16, 2021 at 6:06 PM Michael Thomas  wrote:
> >
> >
> > Basically are there places that you can't get allocations? If so, what
> > is happening?
>
> isn't the answer to this:
>   "All except AFRNic announced their pools were empty."
>
> and:
>   "you can go to the transfer market and arrange transfer of ips for
> pesos (and the proper RIR golden ticketry of course)"

It depends on if you already have IPv4, and how much, and when you
last got it. If you have none and can make a reasoned case, you can
get up to a /23 in APNIC. If you have some but haven't yet got your
final /8 you may qualify for up to a /23. If you already got your
final /23 you can't apply for more.

Open market is not the same as "has run out" -the conditionality on
who gets the small amounts which remain, do not mean nothing remains.

Re: dumb question: are any of the RIR's out of IPv4 addresses?

[George Michaelson]

APNIC continues to have a final /8 policy and can allocate or assign
up to a /23 to new entrants from its holdings.  APNIC reclaims unused
IP addresses.

https://www.apnic.net/manage-ip/ipv4-exhaustion/

On Wed, Feb 17, 2021 at 9:15 AM Jennifer Sims  wrote:
>
> Pretty sure APNIC is out of addresses.
>
> It's now a matter of "when someone sells a block, you buy a block".
>
> On Wed, Feb 17, 2021 at 10:07 AM Michael Thomas  wrote:
>>
>>
>> Basically are there places that you can't get allocations? If so, what
>> is happening?
>>
>> Mike
>>

Re: Texas internet connectivity declining due to blackouts

[Sabri Berisha]

- On Feb 16, 2021, at 6:28 AM, Michael Thomas m...@mtcc.com wrote:

> We use propane. It's less dense energy-wise than gasoline, but it's
> really easy to switch over.

Why not use both? Plenty of generators that are dual fuel out there.
Last year I converted my Duramax to dual fuel by replacing the 
carburator. Easy-peasy.

Thanks,

Sabri

Re: dumb question: are any of the RIR's out of IPv4 addresses?

[William Herrin]

On Tue, Feb 16, 2021 at 3:07 PM Michael Thomas  wrote:
> Basically are there places that you can't get allocations?

All of them except possibly Afrinic.

> If so, what
> is happening?

You find someone who has listed their IP addresses for sale via a
broker, buy them, and then request a specified transfer at the RIR
which transfers those addresses to you.

Regards,
Bill Herrin


-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: dumb question: are any of the RIR's out of IPv4 addresses?

[Christopher Morrow]

On Tue, Feb 16, 2021 at 6:06 PM Michael Thomas  wrote:
>
>
> Basically are there places that you can't get allocations? If so, what
> is happening?

isn't the answer to this:
  "All except AFRNic announced their pools were empty."

and:
  "you can go to the transfer market and arrange transfer of ips for
pesos (and the proper RIR golden ticketry of course)"

Re: Famous operational issues

[Simon Lockhart]

On Tue Feb 16, 2021 at 09:33:20PM +0100, J?rg Kost wrote:
> I don't want to classify and rate it, but would name 9/11.
> 
> You can read about the impacts on the list archives and there is also a
> presentation from NANOG '23 online.

For an operational perspective, I was part of the team trying to keep the
BBC website up and running through 9/11...

http://www.slimey.org/bbc_ticket_10083.txt

Simon

Re: dumb question: are any of the RIR's out of IPv4 addresses?

[Jennifer Sims]

Pretty sure APNIC is out of addresses.

It's now a matter of "when someone sells a block, you buy a block".

On Wed, Feb 17, 2021 at 10:07 AM Michael Thomas  wrote:

>
> Basically are there places that you can't get allocations? If so, what
> is happening?
>
> Mike
>
>

Re: Texas internet connectivity declining due to blackouts

[Seth Mattinen]

On 2/16/21 09:49, Michael Thomas wrote:


On 2/16/21 8:50 AM, John Von Essen wrote:
I just assumed most people in Texas have heat pumps- AC in the summer 
and minimal heating in the winter when needed. When the entire state 
gets a deep freeze, everybody is running those heat pumps non-stop, 
and the generation capacity simply wasn’t there. i.e. coal or natural 
gas plants have some turbines offline, etc.,. in the winter because 
historically power use is much much less. The odd thing is its been 
days now, those plants should be able to ramp back up to capacity - 
but clearly they haven’t. Blaming this on wind turbines is BS. In 
fact, if it weren’t for so many people in Texas with grid-tie solar 
systems, the situation would be even worse.


You'd think that mid-summer Texas chews a lot more peak capacity than 
the middle of winter. Plus I would think a lot of Texas uses natural gas 
for heat rather than electricity further mitigating its effect on the grid.




The difference is that in extreme cold heat pump systems are likely 
switching on emergency heat (i.e. plain old resistance heaters) when the 
compressor alone can no longer keep up with call for heat demand, which 
requires significantly more power. That's never happening in the summer, 
which is only ever running the compressor.

dumb question: are any of the RIR's out of IPv4 addresses?

[Michael Thomas]

Basically are there places that you can't get allocations? If so, what 
is happening?


Mike

Re: Famous operational issues

[Paul Ebersman]

jlewis> This reminds me of one of the Sprint CO's we were colo'd in.

Ah, Sprint. Nothing like using your railroad to run phone lines...
Our routers in San Jose colo were black from the soot of the trains.

Fondly remember a major Sprint outage in the early 90s. All our data
circuits in the southeast went down at once and there were major voice
outages in the entire southeast.

Turns out a storm caused a mudslide which in turn derailed a train
carrying toxic waste, resulting in a wave of 6-10' of toxic mud taking
out the Spring voice pop for the whole southeast, because it was
conveniently located right on said railroad tracks.

We were a big enough customer that PLSC in Atlanta gave us the real
story when we asked for an ETA on repair. They couldn't give us one
immediately until the HAZMAT crew let them in. Turned out to be a total
loss of all gear.

They yanked every tech east of the Misssissippi and a 7ESS was Fedex
overnighted (stolen from some customer in the middle east?) and they had
to rebuild everything.

Was down less than 10 days. Good times.

Re: Famous operational issues

[scott]

On 2/16/2021 9:37 AM, John Kristoff wrote:

I'd suggest the AS 7007 event is perhaps the most notorious and 
likely to top many lists including mine. 




AS7007 is how I found NANOG.  We (Digital Island; first job out
of college) were in 10-20 countries around the planet at the time.
All of them wentdown while we were in cisco training.  I kept
interrupting the class andtelling my manager "everything's down!
We need to stop the training and get on it!"  We didn't because I
was new and no onebelieved that much could go down all at once.
They assumed it was a monitoring glitch.So, the training
continued for a while until very senior engineers got involved.
One of the senior guys said something to the effect of "yeah, it's
all over NANOG."  I said what is NANOG?  I signed upfor the list
and many of you have had to listen to me ever since... ;)

scott

Re: Famous operational issues

[Pierre Emeriaud]

Le mar. 16 févr. 2021 à 21:03, Job Snijders via NANOG
 a écrit :
>
> https://labs.ripe.net/Members/erik/ripe-ncc-and-duke-university-bgp-experiment/
>
> The experiment triggered a bug in some Cisco router models: affected
> Ciscos would corrupt this specific BGP announcement ** ON OUTBOUND **.
> Any peers of such Ciscos receiving this BGP update, would (according to
> then current RFCs) consider the BGP UPDATE corrupted, and would
> subsequently tear down the BGP sessions with the Ciscos. Because the
> corruption was not detected by the Ciscos themselves, whenever the
> sessions would come back online again they'd reannounce the corrupted
> update, causing a session tear down. Bounce ... Bounce ... Bounce ... at
> global scale in both IBGP and EBGP! :-)

In a similar fashion, a network I know had a massive outage when a
failing linecard corrupted is-is lsps, triggering a flood of purges
and taking down the whole backbone.

This was pre-rfc6232, so you can guess that resolving the issue was a real PITA.

This kind of outages fuels my netops nightmares.

Re: [EXTERNAL] Re: Famous operational issues

[Compton, Rich A]

There was the outage in 2014 when we got to 512K routes.  
http://www.bgpmon.net/what-caused-todays-internet-hiccup/


On 2/16/21, 1:04 PM, "NANOG on behalf of Job Snijders via NANOG" 
 
wrote:

CAUTION: The e-mail below is from an external source. Please exercise 
caution before opening attachments, clicking links, or following guidance.

On Tue, Feb 16, 2021 at 01:37:35PM -0600, John Kristoff wrote:
> I'd like to start a thread about the most famous and widespread Internet
> operational issues, outages or implementation incompatibilities you
> have seen.
> 
> Which examples would make up your top three?

This was a fantastic outage, one could really feel the tremors into the
far corners of the BGP default-free zone:


https://labs.ripe.net/Members/erik/ripe-ncc-and-duke-university-bgp-experiment/

The experiment triggered a bug in some Cisco router models: affected
Ciscos would corrupt this specific BGP announcement ** ON OUTBOUND **.
Any peers of such Ciscos receiving this BGP update, would (according to
then current RFCs) consider the BGP UPDATE corrupted, and would
subsequently tear down the BGP sessions with the Ciscos. Because the
corruption was not detected by the Ciscos themselves, whenever the
sessions would come back online again they'd reannounce the corrupted
update, causing a session tear down. Bounce ... Bounce ... Bounce ... at
global scale in both IBGP and EBGP! :-)

Luckily the industry took these, and many other lessons to heart: in
2015 the IETF published RFC 7606 ("Revised Error Handling for BGP UPDATE
Messages") which specifices far more robust behaviour for BGP speakers.

Kind regards,

Job


E-MAIL CONFIDENTIALITY NOTICE: 
The contents of this e-mail message and any attachments are intended solely for 
the addressee(s) and may contain confidential and/or legally privileged 
information. If you are not the intended recipient of this message or if this 
message has been addressed to you in error, please immediately alert the sender 
by reply e-mail and then delete this message and any attachments. If you are 
not the intended recipient, you are notified that any use, dissemination, 
distribution, copying, or storage of this message or any attachment is strictly 
prohibited.

Re: Famous operational issues

[Sean Donelan]

Biggest internet operational SUCCESS

1. Secure Shell (SSH) replaced TELNET. Nearly eliminated an entire class 
of security problems on the Internet.  But then HTTP took over everything, 
so a good news/bad news.


2. Internet worms massively reduced by changed default configurations 
and default firewalls (Windows XP proved defaults could be changed). Still 
need to work on DDOS amplification.


3. Head of Line blocking in IX switches (although I miss Stephen Stuart 
saying "I'm Sorry" at every NANOG for a decade). Was a huge problem, which 
is a non-problem now.


4. Classless Inter-Domain Routing and BGP4 changed how Internet routing 
worked across the entire backbone, and it worked!  Vince Fuller et al 
rebuilt the aircraft in flight, without crashing.


5. Y2K was a huge suggess because a lot of people fixed things ahead time, 
and almost nothing crashed (other than the National Security Agency's 
internal systems :-).  I'll be retired before Y2038, so that's someone 
else's problem.

Re: Texas internet connectivity declining due to blackouts

[William Herrin]

On Mon, Feb 15, 2021 at 8:18 PM Robert Jacobs 
wrote:

> How about letting us Texans have more natural gas power plants or even let
> the gas be delivered to the plants we have so they can provide more power
> in an emergency. Did not help that 20% of our power is now wind which of
> course in an ice storm like we are having is shut off... Lots of issues and
> plenty of politics involved here..
>

CNN reports that some natural gas and oil plants as well as one nuclear
plant have gone offline because the water source they depend on froze.

"Natural gas and coal-fired power plants need water to stay online. Yet
those water facilities froze in the cold temperatures"

https://www.cnn.com/2021/02/16/business/texas-power-energy-nightmare/index.html

Regards,
Bill Herrin

-- 
William Herrin
b...@herrin.us

https://bill.herrin.us/

Re: Famous operational issues

[Jon Lewis]

On Tue, 16 Feb 2021, Sabri Berisha wrote:


- On Feb 16, 2021, at 2:08 PM, Jared Mauch ja...@puck.nether.net wrote:

Hi,


I was thinking about how we need a war stories nanog track. My favorite was
being on call when the router was stolen.


Wait... what? I would love to listen to that call between you and your manager.

But, here is one for you then. I was once called to a POP where one of our main
routers was down. Due to political reasons, my access had been revoked. My
manager told me to do whatever I needed to do to fix the problem, he would cover
my behind. I did, and I "gently" removed the door. My manager held word.


This reminds me of one of the Sprint CO's we were colo'd in.  Access to 
the CLEC colo area was via a back door through the Men's room!  One 
weekend, I had to make the drive to that site to deal with an access 
server issue, and I found they'd locked the back door to the Men's room 
from the colo floor side, so no access.  Using supplies I found inside the 
CO, I managed open the locked door and get to our gear.  That route, being 
our only access route was probably some kind of violation.  Not all of our 
techs were guys.


While we never had a router stolen, we did have a flash card stolen from 
one of our routers in a WCOM colo facility (most customers in open relay 
racks).  It was right after they'd upgraded the doors to the colo area 
from simplex locks to card access.  I was pissed for quite some time that 
WCOM knew who was in there (due to the card access system), but refused to 
tell us.  I figured it was probably one of their own people.


--
 Jon Lewis, MCP :)   |  I route
 StackPath, Sr. Neteng   |  therefore you are
_ http://www.lewis.org/~jlewis/pgp for PGP public key_

Re: Texas internet connectivity declining due to blackouts

[Carsten Bormann]

On 16. Feb 2021, at 16:40, Yang Yu  wrote:
> 
> You can find ERCOT Operations 
> Messageshttp://www.ercot.com/services/comm/mkt_notices/opsmessages

No, I can’t.
(OK, with a handy VPN, I do get access. )

Grüße, Carsten


Access Denied
Error 16
www.ercot.com
2021-02-16 22:12:17 UTC
If you believe you have a valid business reason for accessing ERCOT resources, 
please contact the ERCOT HelpDesk at 512-248-6800 or 1-866-870-8124 (USA) or 
helpd...@ercot.com.
Please provide the HelpDesk with the information supplied below.
Your IP: 80.137.168.40
Error code: 16
 > This request was blocked by the security rules
 
What happened?
This request was blocked by the security rules
Your IP: 80.137.168.40
Proxy IP: 185.11.125.144 (ID 10535-100)
Incident ID: 535000310029634434-105306125617268682

Re: Cross country latency on 3356?

[JASON BOTHE via NANOG]

Lots of amp sites down here currently and with travel impossible and severely 
limited, they’re struggling to restore. It’s not a good situation currently. 

> On Feb 16, 2021, at 14:18, David Hubbard  
> wrote:
> 
> 
> Curious if anyone is seeing issues with 3356 cross country, particularly 
> Orlando-LA?  I have to assume they’re having issues in Texas, so perhaps too 
> much capacity has been lost and it’s overloading what is functioning?
>  
> David

Re: Famous operational issues

[Sabri Berisha]

- On Feb 16, 2021, at 2:08 PM, Jared Mauch ja...@puck.nether.net wrote:

Hi,

> I was thinking about how we need a war stories nanog track. My favorite was
> being on call when the router was stolen.

Wait... what? I would love to listen to that call between you and your manager.

But, here is one for you then. I was once called to a POP where one of our main
routers was down. Due to political reasons, my access had been revoked. My 
manager told me to do whatever I needed to do to fix the problem, he would cover
my behind. I did, and I "gently" removed the door. My manager held word.

Another interesting one: entering a pop to find it flooded. Luckily there were
raised floors with only fiber underneath the floor panels. The NOC ignored the
warnings because "it was impossible for water to enter the building as it was
not raining". Yeah, but water pipes do burst from time to time.

But my favorite was pressing an undocumented combination of keys on a fire
alarm system which set off the Inergen protection without warning, immediately.
The noise and pressure of all that air entering the datacenter space with me
still in it is something I will never forget. Similar to the response of my
manager who, instead of asking me if I was ok, decided to try and light a piece
of paper. "Oh wow, it does work, I can't set anything on fire".

All if this was, obviously, in the late 1990s and early 2000s. These days,
things are -slightly- more professional.

Thanks,

Sabri

Re: Famous operational issues

[Jared Mauch]

I was thinking about how we need a war stories nanog track. My favorite was 
being on call when the router was stolen. 

Sent from my TI-99/4a

> On Feb 16, 2021, at 2:40 PM, John Kristoff  wrote:
> 
> Friends,
> 
> I'd like to start a thread about the most famous and widespread Internet
> operational issues, outages or implementation incompatibilities you
> have seen.
> 
> Which examples would make up your top three?
> 
> To get things started, I'd suggest the AS 7007 event is perhaps  the
> most notorious and likely to top many lists including mine.  So if
> that is one for you I'm asking for just two more.
> 
> I'm particularly interested in this as the first step in developing a
> future NANOG session.  I'd be particularly interested in any issues
> that also identify key individuals that might still be around and
> interested in participating in a retrospective.  I already have someone
> that is willing to talk about AS 7007, which shouldn't be hard to guess
> who.
> 
> Thanks in advance for your suggestions,
> 
> John

Re: Famous operational issues

[Justin Streiner]

Would this also extend to intentional actions that may have had unintended
consequences, such as provider A intentionally de-peering provider B, or
the monopoly telco for $country cutting itself off from the rest of the
global Internet for various reasons (technical, political, or otherwise)?

That said, I'd still have to stick with AS7007, the Baltimore tunnel fire,
and 9/11 as the most prominent examples of widespread issues/outages and
how those issues were addressed.

Honorable mention: $vendor BGP bugs, either due to $vendor ignoring the
relevant RFCs, implementing them incorrectly, or an outage exposed a design
flaw that the RFCs didn't catch.  Too many of those to list here :)

jms

On Tue, Feb 16, 2021 at 2:37 PM John Kristoff  wrote:

> Friends,
>
> I'd like to start a thread about the most famous and widespread Internet
> operational issues, outages or implementation incompatibilities you
> have seen.
>
> Which examples would make up your top three?
>
> To get things started, I'd suggest the AS 7007 event is perhaps  the
> most notorious and likely to top many lists including mine.  So if
> that is one for you I'm asking for just two more.
>
> I'm particularly interested in this as the first step in developing a
> future NANOG session.  I'd be particularly interested in any issues
> that also identify key individuals that might still be around and
> interested in participating in a retrospective.  I already have someone
> that is willing to talk about AS 7007, which shouldn't be hard to guess
> who.
>
> Thanks in advance for your suggestions,
>
> John
>

Re: Famous operational issues

[Jörg Kost]

Oh well, MCI in 1999 was all about…
https://www.youtube.com/watch?v=7iM5nFNUG4U

On 16 Feb 2021, at 22:28, Sean Donelan wrote:


Since you said operational issues, instead of just outage...

How about MCI Worldcom's 10-day operational disaster in 1999.


http://www.cnn.com/TECH/computing/9908/23/network.nono.idg/
How not to handle a network outage

[...]
MCI WorldCom issued an alert to its sales force, which was given the 
option to deliver a notice to customers by e-mail, hand delivery or 
telephone – or not at all. After a deafening silence from company 
executives on the 10-day network outage, MCI WorldCom CEO Bernie 
Ebbers finally took the podium to discuss the situation. How did he 
explain the failure, and reassure customers that the network would not 
suffer such a failure in the future? He didn't. Instead, he blamed 
Lucent.

[...]

Re: Famous operational issues

[Todd Underwood]

There are all the hilarious leaks and blocks.

Pakistan blocks youtube and the announcement leaks internet-wide.
Turk telecom (AS9121 IIRC) leaks a full table out one of their providers.

So many routing level incidents they're probably not even interesting any
more,  I suppose.

The huge power outages in the US northeast in 2003 (
https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.183.998=rep1=pdf)
were pretty decent.



On Tue, Feb 16, 2021 at 4:02 PM Damian Menscher via NANOG 
wrote:

> https://en.wikipedia.org/wiki/SQL_Slammer was interesting in that it was
> an application-layer issue that affected the network layer.
>
> Damian
>
> On Tue, Feb 16, 2021 at 11:37 AM John Kristoff  wrote:
>
>> Friends,
>>
>> I'd like to start a thread about the most famous and widespread Internet
>> operational issues, outages or implementation incompatibilities you
>> have seen.
>>
>> Which examples would make up your top three?
>>
>> To get things started, I'd suggest the AS 7007 event is perhaps  the
>> most notorious and likely to top many lists including mine.  So if
>> that is one for you I'm asking for just two more.
>>
>> I'm particularly interested in this as the first step in developing a
>> future NANOG session.  I'd be particularly interested in any issues
>> that also identify key individuals that might still be around and
>> interested in participating in a retrospective.  I already have someone
>> that is willing to talk about AS 7007, which shouldn't be hard to guess
>> who.
>>
>> Thanks in advance for your suggestions,
>>
>> John
>>
>

Re: Famous operational issues

[Sean Donelan]

Since you said operational issues, instead of just outage...

How about MCI Worldcom's 10-day operational disaster in 1999.


http://www.cnn.com/TECH/computing/9908/23/network.nono.idg/
How not to handle a network outage

[...]
MCI WorldCom issued an alert to its sales force, which was given the 
option to deliver a notice to customers by e-mail, hand delivery or 
telephone – or not at all. After a deafening silence from company 
executives on the 10-day network outage, MCI WorldCom CEO Bernie Ebbers 
finally took the podium to discuss the situation. How did he explain the 
failure, and reassure customers that the network would not suffer such a 
failure in the future? He didn't. Instead, he blamed Lucent.

[...]

Re: Texas internet connectivity declining due to blackouts

[Chris Boyd]

> On Feb 16, 2021, at 11:51 AM, Michael Thomas  wrote:
> 
> You'd think that mid-summer Texas chews a lot more peak capacity than the 
> middle of winter. Plus I would think a lot of Texas uses natural gas for heat 
> rather than electricity further mitigating its effect on the grid.
> 
> Mike

The eia.gov site shows it to be about a 50/50 split between natural gas and 
electric heating. Propane fills in a few more percent. Yes, the grid does get 
quite strained in the summer from AC use. 

—Chris, from Austin

Re: Famous operational issues

[Damian Menscher via NANOG]

https://en.wikipedia.org/wiki/SQL_Slammer was interesting in that it was an
application-layer issue that affected the network layer.

Damian

On Tue, Feb 16, 2021 at 11:37 AM John Kristoff  wrote:

> Friends,
>
> I'd like to start a thread about the most famous and widespread Internet
> operational issues, outages or implementation incompatibilities you
> have seen.
>
> Which examples would make up your top three?
>
> To get things started, I'd suggest the AS 7007 event is perhaps  the
> most notorious and likely to top many lists including mine.  So if
> that is one for you I'm asking for just two more.
>
> I'm particularly interested in this as the first step in developing a
> future NANOG session.  I'd be particularly interested in any issues
> that also identify key individuals that might still be around and
> interested in participating in a retrospective.  I already have someone
> that is willing to talk about AS 7007, which shouldn't be hard to guess
> who.
>
> Thanks in advance for your suggestions,
>
> John
>

Re: Famous operational issues

[Randy Bush]

> actually, the 129/8 incident

a friend pointed out that it was the 128/9 incident

> but folk tend not to remember it

qed, eh?  :)

Re: Famous operational issues

[Randy Bush]

actually, the 129/8 incident was as damaging as 7007, but folk tend not
to remember it; maybe because it was a bit embarrassing

and the baltimore tunnel is a gift that gave a few times

and the quake/mudslides off taiwan

the tohoku quake was also fun, in some sense of the word

but the list of really damaging wet glass cuts is long

Re: Famous operational issues

[Jörg Kost]

Hi,

I don't want to classify and rate it, but would name 9/11.

You can read about the impacts on the list archives and there is also a 
presentation from NANOG '23 online.


Regards
Jörg

On 16 Feb 2021, at 20:37, John Kristoff wrote:


Friends,

I'd like to start a thread about the most famous and widespread 
Internet

operational issues, outages or implementation incompatibilities you
have seen.

Which examples would make up your top three?

Cross country latency on 3356?

[David Hubbard]

Curious if anyone is seeing issues with 3356 cross country, particularly 
Orlando-LA?  I have to assume they’re having issues in Texas, so perhaps too 
much capacity has been lost and it’s overloading what is functioning?

David

Re: Famous operational issues

[Mikael Abrahamsson via NANOG]

On Tue, 16 Feb 2021, John Kristoff wrote:


Friends,

I'd like to start a thread about the most famous and widespread Internet
operational issues, outages or implementation incompatibilities you
have seen.

Which examples would make up your top three?


https://blogs.oracle.com/internetintelligence/longer-is-not-always-better

--
Mikael Abrahamssonemail: swm...@swm.pp.se

Re: Famous operational issues

[Job Snijders via NANOG]

On Tue, Feb 16, 2021 at 01:37:35PM -0600, John Kristoff wrote:
> I'd like to start a thread about the most famous and widespread Internet
> operational issues, outages or implementation incompatibilities you
> have seen.
> 
> Which examples would make up your top three?

This was a fantastic outage, one could really feel the tremors into the
far corners of the BGP default-free zone:

https://labs.ripe.net/Members/erik/ripe-ncc-and-duke-university-bgp-experiment/

The experiment triggered a bug in some Cisco router models: affected
Ciscos would corrupt this specific BGP announcement ** ON OUTBOUND **.
Any peers of such Ciscos receiving this BGP update, would (according to
then current RFCs) consider the BGP UPDATE corrupted, and would
subsequently tear down the BGP sessions with the Ciscos. Because the
corruption was not detected by the Ciscos themselves, whenever the
sessions would come back online again they'd reannounce the corrupted
update, causing a session tear down. Bounce ... Bounce ... Bounce ... at
global scale in both IBGP and EBGP! :-)

Luckily the industry took these, and many other lessons to heart: in
2015 the IETF published RFC 7606 ("Revised Error Handling for BGP UPDATE
Messages") which specifices far more robust behaviour for BGP speakers.

Kind regards,

Job

Famous operational issues

[John Kristoff]

Friends,

I'd like to start a thread about the most famous and widespread Internet
operational issues, outages or implementation incompatibilities you
have seen.

Which examples would make up your top three?

To get things started, I'd suggest the AS 7007 event is perhaps  the
most notorious and likely to top many lists including mine.  So if
that is one for you I'm asking for just two more.

I'm particularly interested in this as the first step in developing a
future NANOG session.  I'd be particularly interested in any issues
that also identify key individuals that might still be around and
interested in participating in a retrospective.  I already have someone
that is willing to talk about AS 7007, which shouldn't be hard to guess
who.

Thanks in advance for your suggestions,

John

Re: Texas internet connectivity declining due to blackouts

[Rich Kulawiec]

On Tue, Feb 16, 2021 at 04:17:15AM +, Robert Jacobs wrote:
> How about letting us Texans have more natural gas power plants or even
> let the gas be delivered to the plants we have so they can provide more
> power in an emergency.  Did not help that 20% of our power is now wind
> which of course in an ice storm like we are having is shut off... Lots
> of issues and plenty of politics involved here..

First, wind-generation-is-responsible-for-this is a canard that's
already been debunked elsewhere in this thread.

Second, wind generation works just fine all winter in places like
Quebec and the Alps because they design, build, and operate it to.
Various de-icing solutions have been available for years, and market
competition is continuously making them better and cheaper.

Third, trying to slap a fossil fuel band-aid on a problem whose root
cause is...wait for it...fossil fuels...while certainly a tempting option,
is not a viable long-term solution.

---rsk

Google Nest camera contact request

[Chuck Church]

If anyone has access to a Google Nest engineer could you pass their info to
me offline?  Going through tech support didn't get me anywhere.  We're
seeing that their cameras and the cloud servers they home back to
(oculus-xxx.dropcam.com) are using TLS 1.0 and no server name in the certs,
which our firewalls are dropping and our security folks are reluctant to
relax rules to make work.  Hoping for a status on when modern security
communications might be used on these.

 

Thanks,

 

Chuck

Re: Texas internet connectivity declining due to blackouts

[Sabri Berisha]

- On Feb 16, 2021, at 5:01 AM, Sean Donelan s...@donelan.com wrote:

> On Tue, 16 Feb 2021, Rod Beck wrote:
>> Are the power lines buried like in Europe where I live?

They are not buried everywhere. They are buried in most western EU
countries perhaps. But I invite you to go to Ferizaj, Kosovo, for example.
 
> In California, they use rolling blackouts BEFORE wildfires to prevent
> power line sparking causing wildfires. Not because of damage to the
> outside plant. In Texas, they use rolling blackouts because they didn't
> have enough generation capacity online.

I do remember last September being threatened with rolling power outages
as a result of the lack of capacity. 

Check this article in the Mercury News, for example: 
https://www.mercurynews.com/2020/09/06/california-grid-managers-watching-closely-as-weather-presents-power-outage-threats/

Thanks,

Sabri

Re: Texas internet connectivity declining due to blackouts

[Ishmael Rufus]

Maybe Texas can learn from its Northern neighbors.
[image: image.png]

On Mon, Feb 15, 2021 at 10:19 PM Robert Jacobs 
wrote:

> How about letting us Texans have more natural gas power plants or even let
> the gas be delivered to the plants we have so they can provide more power
> in an emergency. Did not help that 20% of our power is now wind which of
> course in an ice storm like we are having is shut off... Lots of issues and
> plenty of politics involved here..
>
> Robert Jacobs​
>  |  Data Center Manager
> 
> Direct:  *832-615-7742* <832-615-7742>
> Mobile:  *281-830-2092* <281-830-2092>
> Main:  832‑615‑8000
> Fax:  *713-510-1650*
> 5959 Corporate Dr. Suite 3300; Houston, TX 77036
> [image: Facebook] 
> [image: LinkedIn] 
> [image: Twitter] 
>  A Certified Woman‑Owned Business
> 24x7x365 Customer Support: 832-615-8000 | supp...@pslightwave.com
>
> ​This electronic message contains information from PS Lightwave which may
> be privileged and confidential. The information is intended to be for the
> use of individual(s) or entity named above. If you are not the intended
> recipient, any disclosure, copying, distribution or use of the contents of
> this information is prohibited. If you have received this electronic
> message in error, please notify me by telephone or e-mail immediately.
> -Original Message-
> From: NANOG  On Behalf
> Of Mark Tinka
> Sent: Monday, February 15, 2021 10:06 PM
> To: nanog@nanog.org
> Subject: Re: Texas internet connectivity declining due to blackouts
>
>
>
> On 2/16/21 04:14, Sean Donelan wrote:
> >
> > Poweroutage.us posted a terrific map, showing the jurisdictional
> > borders of the Texas power outages versus the storm related power
> > outages elsewhere in the country.
> >
> > https://twitter.com/PowerOutage_us/status/1361493394070118402
> >
> >
> > Sometimes infrastructure planning failures are not due to "natural
> > hazards."
>
> I suppose having some kind of home backup solution wouldn't be too bad
> right now, even though you may still not get access to services. But at
> least, you can brew some coffee, and charge your pulse oximetre.
>
> Mark.
>
>

Re: Texas internet connectivity declining due to blackouts

[Miles Fidelman]

Last time I looked - admittedly a while - Texas made it VERY difficult 
for municipalities to set up broadband utilities, even in areas where no 
commercial player was interested.  Maybe that's catching up to them.


Miles Fidelman

Brandon Svec wrote:
Mismanagement and poor planning are primarily to blame.  One can't 
just blame the weather.  We know weather will be bad and have extreme 
variations.  I am sure Texas politicians are considering what they 
could have done better right now.. 
https://twitter.com/blkahn/status/1361682089310052354

*Brandon *



On Tue, Feb 16, 2021 at 8:53 AM John Von Essen > wrote:


I just assumed most people in Texas have heat pumps- AC in the
summer and minimal heating in the winter when needed. When the
entire state gets a deep freeze, everybody is running those heat
pumps non-stop, and the generation capacity simply wasn’t there.
i.e. coal or natural gas plants have some turbines offline, etc.,.
in the winter because historically power use is much much less.
The odd thing is its been days now, those plants should be able to
ramp back up to capacity - but clearly they haven’t. Blaming this
on wind turbines is BS. In fact, if it weren’t for so many people
in Texas with grid-tie solar systems, the situation would be even
worse.

And of course, the real issue is Texas’ closed grid - any other
state could pull in more power from neighbors.

-John


On Feb 15, 2021, at 11:34 PM, Cory Sell via NANOG
mailto:nanog@nanog.org>> wrote:

Ercot has already released actual documentation of the outputs.
Wind is NOT the biggest loss here. Even if wind was operating at
100% capacity, we’d be in the same boat due to gas and fossil
fuel-related generation being decimated. Estimated 4GW lost for
wind doesn’t make up for the 30GW+ estimated being lost from
fossil fuels.

I only interject because people are already pointing their
fingers at renewables being the cause here and trying to pawn off
the blame to wind/solar to further their agendas to reduce
renewable energy R and adoption. Sure, wind isn’t perfect, but
looks like solution relied on failed in a massive way.

Sent from ProtonMail Mobile


On Mon, Feb 15, 2021 at 10:17 PM, Robert Jacobs
mailto:rjac...@pslightwave.com>> wrote:

How about letting us Texans have more natural gas power plants
or even let the gas be delivered to the plants we have so they
can provide more power in an emergency. Did not help that 20% of
our power is now wind which of course in an ice storm like we
are having is shut off... Lots of issues and plenty of politics
involved here..

Robert Jacobs

 |  Data Center Manager


Direct: *832-615-7742* 
Mobile: *281-830-2092* 
Main:   832‑615‑8000
Fax:*713-510-1650*

5959 Corporate Dr. Suite 3300; Houston, TX 77036
Facebook 


LinkedIn 


Twitter 

 A Certified Woman‑Owned 
Business

24x7x365 Customer Support: 832-615-8000 |
supp...@pslightwave.com 

This electronic message contains information from PS Lightwave
which may be privileged and confidential. The information is
intended to be for the use of individual(s) or entity named
above. If you are not the intended recipient, any disclosure,
copying, distribution or use of the contents of this information
is prohibited. If you have received this electronic message in
error, please notify me by telephone or e-mail immediately.

-Original Message-
From: NANOG mailto:nanog-bounces+rjacobs=pslightwave@nanog.org>> On
Behalf Of Mark Tinka
Sent: Monday, February 15, 2021 10:06 PM
To: nanog@nanog.org 
Subject: Re: Texas internet connectivity declining due to blackouts



On 2/16/21 04:14, Sean Donelan wrote:
>
> Poweroutage.us  posted a terrific map,
showing the jurisdictional
> borders of the Texas power outages versus the storm related power
> outages elsewhere in the country.
>
> https://twitter.com/PowerOutage_us/status/1361493394070118402
>
>
> Sometimes infrastructure planning failures are not due to
"natural
> hazards."

I suppose having some kind of home backup solution wouldn't be
too bad right now, even though you may still not get access to
services. But at least, you can brew some coffee, and charge
your pulse oximetre.

Mark.









--
In theory, there is no difference between theory and practice.
In practice, there is.   Yogi Berra

Theory is when you know everything but nothing works.

Re: Texas internet connectivity declining due to blackouts

[Michael Thomas]

On 2/16/21 8:50 AM, John Von Essen wrote:
I just assumed most people in Texas have heat pumps- AC in the summer 
and minimal heating in the winter when needed. When the entire state 
gets a deep freeze, everybody is running those heat pumps non-stop, 
and the generation capacity simply wasn’t there. i.e. coal or natural 
gas plants have some turbines offline, etc.,. in the winter because 
historically power use is much much less. The odd thing is its been 
days now, those plants should be able to ramp back up to capacity - 
but clearly they haven’t. Blaming this on wind turbines is BS. In 
fact, if it weren’t for so many people in Texas with grid-tie solar 
systems, the situation would be even worse.


You'd think that mid-summer Texas chews a lot more peak capacity than 
the middle of winter. Plus I would think a lot of Texas uses natural gas 
for heat rather than electricity further mitigating its effect on the grid.


Mike

Re: Texas internet connectivity declining due to blackouts

[Brandon Svec]

Mismanagement and poor planning are primarily to blame.  One can't just
blame the weather.  We know weather will be bad and have extreme
variations.  I am sure Texas politicians are considering what they could
have done better right now..
https://twitter.com/blkahn/status/1361682089310052354
*Brandon *


On Tue, Feb 16, 2021 at 8:53 AM John Von Essen  wrote:

> I just assumed most people in Texas have heat pumps- AC in the summer and
> minimal heating in the winter when needed. When the entire state gets a
> deep freeze, everybody is running those heat pumps non-stop, and the
> generation capacity simply wasn’t there. i.e. coal or natural gas plants
> have some turbines offline, etc.,. in the winter because historically power
> use is much much less. The odd thing is its been days now, those plants
> should be able to ramp back up to capacity - but clearly they haven’t.
> Blaming this on wind turbines is BS. In fact, if it weren’t for so many
> people in Texas with grid-tie solar systems, the situation would be even
> worse.
>
> And of course, the real issue is Texas’ closed grid - any other state
> could pull in more power from neighbors.
>
> -John
>
> On Feb 15, 2021, at 11:34 PM, Cory Sell via NANOG  wrote:
>
> Ercot has already released actual documentation of the outputs. Wind is
> NOT the biggest loss here. Even if wind was operating at 100% capacity,
> we’d be in the same boat due to gas and fossil fuel-related generation
> being decimated. Estimated 4GW lost for wind doesn’t make up for the 30GW+
> estimated being lost from fossil fuels.
>
> I only interject because people are already pointing their fingers at
> renewables being the cause here and trying to pawn off the blame to
> wind/solar to further their agendas to reduce renewable energy R and
> adoption. Sure, wind isn’t perfect, but looks like solution relied on
> failed in a massive way.
>
> Sent from ProtonMail Mobile
>
>
> On Mon, Feb 15, 2021 at 10:17 PM, Robert Jacobs 
> wrote:
>
> How about letting us Texans have more natural gas power plants or even let
> the gas be delivered to the plants we have so they can provide more power
> in an emergency. Did not help that 20% of our power is now wind which of
> course in an ice storm like we are having is shut off... Lots of issues and
> plenty of politics involved here..
>
> Robert Jacobs
>  |  Data Center Manager
> 
> Direct:  *832-615-7742* <832-615-7742>
> Mobile:  *281-830-2092* <281-830-2092>
> Main:  832‑615‑8000
> Fax:  *713-510-1650*
> 5959 Corporate Dr. Suite 3300; Houston, TX 77036
> [image: Facebook] 
> [image: LinkedIn] 
> [image: Twitter] 
>  A Certified Woman‑Owned Business
> 24x7x365 Customer Support: 832-615-8000 | supp...@pslightwave.com
>
> This electronic message contains information from PS Lightwave which may
> be privileged and confidential. The information is intended to be for the
> use of individual(s) or entity named above. If you are not the intended
> recipient, any disclosure, copying, distribution or use of the contents of
> this information is prohibited. If you have received this electronic
> message in error, please notify me by telephone or e-mail immediately.
> -Original Message-
> From: NANOG  On Behalf
> Of Mark Tinka
> Sent: Monday, February 15, 2021 10:06 PM
> To: nanog@nanog.org
> Subject: Re: Texas internet connectivity declining due to blackouts
>
>
>
> On 2/16/21 04:14, Sean Donelan wrote:
> >
> > Poweroutage.us posted a terrific map, showing the jurisdictional
> > borders of the Texas power outages versus the storm related power
> > outages elsewhere in the country.
> >
> > https://twitter.com/PowerOutage_us/status/1361493394070118402
> >
> >
> > Sometimes infrastructure planning failures are not due to "natural
> > hazards."
>
> I suppose having some kind of home backup solution wouldn't be too bad
> right now, even though you may still not get access to services. But at
> least, you can brew some coffee, and charge your pulse oximetre.
>
> Mark.
>
>
>
>
>

Re: Texas internet connectivity declining due to blackouts

[John Von Essen]

I just assumed most people in Texas have heat pumps- AC in the summer and 
minimal heating in the winter when needed. When the entire state gets a deep 
freeze, everybody is running those heat pumps non-stop, and the generation 
capacity simply wasn’t there. i.e. coal or natural gas plants have some 
turbines offline, etc.,. in the winter because historically power use is much 
much less. The odd thing is its been days now, those plants should be able to 
ramp back up to capacity - but clearly they haven’t. Blaming this on wind 
turbines is BS. In fact, if it weren’t for so many people in Texas with 
grid-tie solar systems, the situation would be even worse. 

And of course, the real issue is Texas’ closed grid - any other state could 
pull in more power from neighbors.

-John

> On Feb 15, 2021, at 11:34 PM, Cory Sell via NANOG  wrote:
> 
> Ercot has already released actual documentation of the outputs. Wind is NOT 
> the biggest loss here. Even if wind was operating at 100% capacity, we’d be 
> in the same boat due to gas and fossil fuel-related generation being 
> decimated. Estimated 4GW lost for wind doesn’t make up for the 30GW+ 
> estimated being lost from fossil fuels. 
> 
> I only interject because people are already pointing their fingers at 
> renewables being the cause here and trying to pawn off the blame to 
> wind/solar to further their agendas to reduce renewable energy R and 
> adoption. Sure, wind isn’t perfect, but looks like solution relied on failed 
> in a massive way.
> 
> Sent from ProtonMail Mobile
> 
> 
> On Mon, Feb 15, 2021 at 10:17 PM, Robert Jacobs  > wrote:
>> 
>> How about letting us Texans have more natural gas power plants or even let 
>> the gas be delivered to the plants we have so they can provide more power in 
>> an emergency. Did not help that 20% of our power is now wind which of course 
>> in an ice storm like we are having is shut off... Lots of issues and plenty 
>> of politics involved here.. 
>> 
>> Robert Jacobs​
>>  |   Data Center Manager
>>  
>> Direct:  832-615-7742 
>> Mobile:  281-830-2092 
>> Main:832‑615‑8000
>> Fax: 713-510-1650 <>
>> 5959 Corporate Dr. Suite 3300; Houston, TX 77036 
>>   
>>    
>>  
>>     A Certified Woman‑Owned 
>> Business 
>> 24x7x365 Customer Support: 832-615-8000 | supp...@pslightwave.com
>> 
>> ​This electronic message contains information from PS Lightwave which may be 
>> privileged and confidential. The information is intended to be for the use 
>> of individual(s) or entity named above. If you are not the intended 
>> recipient, any disclosure, copying, distribution or use of the contents of 
>> this information is prohibited. If you have received this electronic message 
>> in error, please notify me by telephone or e-mail immediately.
>> -Original Message-
>> From: NANOG  On Behalf Of 
>> Mark Tinka
>> Sent: Monday, February 15, 2021 10:06 PM
>> To: nanog@nanog.org
>> Subject: Re: Texas internet connectivity declining due to blackouts
>> 
>> 
>> 
>> On 2/16/21 04:14, Sean Donelan wrote:
>> >
>> > Poweroutage.us posted a terrific map, showing the jurisdictional 
>> > borders of the Texas power outages versus the storm related power 
>> > outages elsewhere in the country.
>> >
>> > https://twitter.com/PowerOutage_us/status/1361493394070118402
>> >
>> >
>> > Sometimes infrastructure planning failures are not due to "natural 
>> > hazards."
>> 
>> I suppose having some kind of home backup solution wouldn't be too bad right 
>> now, even though you may still not get access to services. But at least, you 
>> can brew some coffee, and charge your pulse oximetre.
>> 
>> Mark.
>> 
> 
> 

Re: Texas internet connectivity declining due to blackouts

[Matt Erculiani]

Texas does have DC ties to other grids, there just isn't enough capacity
there to really matter (820 MW total to the Eastern US grid). This is
fairly standard though when you're on your own grid, UK and France have DC
ties as well that run underneath the English Channel.

ERCOT is using a little bit currently:
http://www.ercot.com/content/cdr/html/real_time_system_conditions.html

On Tue, Feb 16, 2021 at 9:17 AM Mike Hammett  wrote:

> Agreed.
>
> Well, or interconnection with other grids that *do* have available
> generation.  :-)
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions 
> 
> 
> 
> 
> Midwest Internet Exchange 
> 
> 
> 
> The Brothers WISP 
> 
> 
> --
> *From: *"Robert DeVita" 
> *To: *"Mike Hammett" , "Rod Beck" <
> rod.b...@unitedcablecompany.com>
> *Cc: *nanog@nanog.org
> *Sent: *Tuesday, February 16, 2021 7:30:50 AM
> *Subject: *Re: Texas internet connectivity declining due to blackouts
>
> What’s going on in Texas has nothing to do with power distribution. It has
> to do with ability to generate power.
>
> Robert DeVita
> Founder & CEO
> Mejeticks
> c. 469-441-8864
> e. radev...@mejeticks.com
> --
> *From:* NANOG  on behalf
> of Mike Hammett 
> *Sent:* Tuesday, February 16, 2021 7:25:12 AM
> *To:* Rod Beck 
> *Cc:* nanog@nanog.org 
> *Subject:* Re: Texas internet connectivity declining due to blackouts
>
> It's cheaper to build 2x, 3x, 4x the aerial plant than to build 1x the
> underground plant.
>
> The actual cost per foot is more like 10x difference, but there are right
> of way, maintenance, etc. costs to factor in as well.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
>
> Midwest Internet Exchange
>
> The Brothers WISP
>
> --
> *From: *"Rod Beck" 
> *To: *"Sean Donelan" , "Mikael Abrahamsson" <
> swm...@swm.pp.se>
> *Cc: *nanog@nanog.org
> *Sent: *Tuesday, February 16, 2021 6:05:41 AM
> *Subject: *Re: Texas internet connectivity declining due to blackouts
>
> Are the power lines buried like in Europe where I live?
>
> I really think using poles is crazy and global warming guarantees enough
> atmospheric turbulence to make it untenable. Florida is moving to bury
> power lines.
>
> --
> *From: *NANOG 
> on behalf of Mikael Abrahamsson via NANOG 
> *Sent: *Tuesday, February 16, 2021 9:06 AM
> *To: *Sean Donelan 
> *Cc: *nanog@nanog.org 
> *Subject: *RE: Texas internet connectivity declining due to blackouts
> On Mon, 15 Feb 2021, Sean Donelan wrote:
>
> > Strange the massive shortages and failures are only in one state.
> >
> > The extreme cold weather extends northwards across many states, which
> aren't
> > reporting rolling blackouts.
>
> https://www.texastribune.org/2011/02/08/texplainer-why-does-texas-have-its-own-power-grid/
>
>
> Going at it alone can be beneficial sometimes, sometimes it's not.
>
> --
> Mikael Abrahamssonemail: swm...@swm.pp.se
>
>
>

-- 
Matt Erculiani
ERCUL-ARIN

Re: Infomart Dallas is on generator

[Matt Erculiani]

This should not come as a surprise. It's not uncommon for Texas DCs to do
this in the Summer to reduce load on the grid during very hot days. They
are often notified a few days in advance and compensated for it beyond fuel
cost and equipment wear & tear. With all the hubbub lately, they knew this
was coming.

It serves 2 purposes: A, a data center can basically be paid to run on
generators, and B, they can control when and how gracefully they transfer
load (can give generators the opportunity to warm up, a luxury they seldom
have) because they may end up losing utility at some point regardless.

- Matt

On Tue, Feb 16, 2021 at 7:31 AM Keith Stokes  wrote:

> Equinix DA-2 reported loads transferred 3-4 a.m.
>
>
>
> --
> *From:* NANOG  on behalf of
> Robert DeVita 
> *Sent:* Monday, February 15, 2021 4:51 PM
> *To:* Eric Kuhnke ; nanog@nanog.org list <
> nanog@nanog.org>
> *Subject:* Re: Infomart Dallas is on generator
>
> Hopefully the other 400mw in Dallas follow their lead.
>
> Robert DeVita
> Founder & CEO
> Mejeticks
> c. 469-441-8864
> e. radev...@mejeticks.com
> --
> *From:* NANOG  on behalf
> of Eric Kuhnke 
> *Sent:* Monday, February 15, 2021 4:10:32 PM
> *To:* nanog@nanog.org list 
> *Subject:* Infomart Dallas is on generator
>
> I have now heard from two reliable sources that Infomart Dallas is
> presently on generator, and is likely to remain so until the cold
> weather/electrical supply emergency in Texas has abated. No network impact
> seen yet.
>
>
>

-- 
Matt Erculiani
ERCUL-ARIN

Re: Texas internet connectivity declining due to blackouts

[Mike Hammett]

I was referring to electrical distribution or transmission. Putting in a 2" 
conduit with some glass in it is a different beast than 34kv or 345kv lines. 
:-) 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Jared Mauch"  
To: "Mike Hammett"  
Cc: "Rod Beck" , nanog@nanog.org 
Sent: Tuesday, February 16, 2021 8:38:11 AM 
Subject: Re: Texas internet connectivity declining due to blackouts 



> On Feb 16, 2021, at 8:25 AM, Mike Hammett  wrote: 
> 
> It's cheaper to build 2x, 3x, 4x the aerial plant than to build 1x the 
> underground plant. 
> 
> The actual cost per foot is more like 10x difference, but there are right of 
> way, maintenance, etc. costs to factor in as well. 
> 

Labor is something in 8x but the permit/engineering cost is usually the same 
per foot, but the make-ready on poles can make underground competitive or more 
like 1.5x when you fully bake the costs. Really depends on pole distances and 
quality. O long-term is lower on underground vs aerial. 

- Jared 

Re: Texas internet connectivity declining due to blackouts

[Mike Hammett]

Agreed. 

Well, or interconnection with other grids that *do* have available generation. 
:-) 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Robert DeVita"  
To: "Mike Hammett" , "Rod Beck" 
 
Cc: nanog@nanog.org 
Sent: Tuesday, February 16, 2021 7:30:50 AM 
Subject: Re: Texas internet connectivity declining due to blackouts 





What’s going on in Texas has nothing to do with power distribution. It has to 
do with ability to generate power. 



Robert DeVita 
Founder & CEO 
Mejeticks 
c. 469-441-8864 
e. radev...@mejeticks.com 

From: NANOG  on behalf of Mike 
Hammett  
Sent: Tuesday, February 16, 2021 7:25:12 AM 
To: Rod Beck  
Cc: nanog@nanog.org  
Subject: Re: Texas internet connectivity declining due to blackouts 


It's cheaper to build 2x, 3x, 4x the aerial plant than to build 1x the 
underground plant. 


The actual cost per foot is more like 10x difference, but there are right of 
way, maintenance, etc. costs to factor in as well. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 



From: "Rod Beck"  
To: "Sean Donelan" , "Mikael Abrahamsson"  
Cc: nanog@nanog.org 
Sent: Tuesday, February 16, 2021 6:05:41 AM 
Subject: Re: Texas internet connectivity declining due to blackouts 


Are the power lines buried like in Europe where I live? 



I really think using poles is crazy and global warming guarantees enough 
atmospheric turbulence to make it untenable. Florida is moving to bury power 
lines. 






From: NANOG  on behalf 
of Mikael Abrahamsson via NANOG  
Sent: Tuesday, February 16, 2021 9:06 AM 
To: Sean Donelan  
Cc: nanog@nanog.org  
Subject: RE: Texas internet connectivity declining due to blackouts 


On Mon, 15 Feb 2021, Sean Donelan wrote: 

> Strange the massive shortages and failures are only in one state. 
> 
> The extreme cold weather extends northwards across many states, which aren't 
> reporting rolling blackouts. 

https://www.texastribune.org/2011/02/08/texplainer-why-does-texas-have-its-own-power-grid/
 

Going at it alone can be beneficial sometimes, sometimes it's not. 

-- 
Mikael Abrahamsson email: swm...@swm.pp.se 

Re: Texas internet connectivity declining due to blackouts

[JASON BOTHE via NANOG]

The professor has it right. Before the state privatized the grid and made 
ERCOT, we never had these problems. Every few years, these private companies 
complain they need a rate hike because they need a grant to ‘beef up’ the 
infrastructure and it’s granted although we seem to keep having this issue. I’m 
certain history will continue to repeat itself. 

> On Feb 16, 2021, at 06:32, John Sage  wrote:
> 
> On 2/16/21 4:22 AM, John Sage wrote:
>>> On 2/15/21 10:02 PM, Mark Tinka wrote:
>>> 
>>> 
 On 2/16/21 07:49, Matthew Petach wrote:
>>> 
 
 Isn't that a result of ERCOT stubbornly refusing to interconnect with the 
 rest of the national grid, out of an irrational fear of coming under 
 federal regulation?
>> Yes. This has been widely documented in numerous articles, both very 
>> recently and previously.
> 
> As one example only, of many:
> 
> "What went wrong with the Texas power grid?"
> 
> Marcy de Luna, Amanda Drane, Houston Chronicle
> 
> Feb. 15, 2021
> Updated: Feb. 15, 2021 9:23 p.m.
> 
> "Dan Woodfin, ERCOT’s senior director of system operations, said the rolling 
> blackouts are taking more power offline for longer periods than ever before. 
> An estimated 34,000 megawatts of power generation — more than a third of the 
> system’s total generating capacity — had been knocked offline by the extreme 
> winter weather amid soaring demand as residents crank up heating systems."
> 
> . . .
> 
> "Ed Hirs, an energy fellow in the Department of Economics at the University 
> of Houston, blamed the failures on the state’s deregulated power system, 
> which doesn’t provide power generators with the returns needed to invest in 
> maintaining and improving power plants.
> 
> “The ERCOT grid has collapsed in exactly the same manner as the old Soviet 
> Union,” said Hirs. “It limped along on underinvestment and neglect until it 
> finally broke under predictable circumstances."
> 
> https://www.houstonchronicle.com/business/energy/article/Wholesale-power-prices-spiking-across-Texas-15951684.php
>  
> 
> 
> - John
> __

Re: Texas internet connectivity declining due to blackouts

[Yang Yu]

On Tue, Feb 16, 2021 at 6:11 AM Rod Beck 
wrote:

> Anyone wants to provide some details on where the system has faltered? It
> is transmission? Or generation? Or just everything in general? 
>


You can find ERCOT Operations Messageshttp://
www.ercot.com/services/comm/mkt_notices/opsmessages
>From what I understand generation/transmission/distribution are all
affected to different degrees.

On Fuel Mix Report: 2021, wind was 25% by GWh for 2021 January, current it
is ~9%

http://www.ercot.com/gridinfo/generation
http://www.ercot.com/content/cdr/html/real_time_system_conditions.html  (DC
Tie is non-synchronize connection to other grids)


>Market Participants that own or operate facilities that are part of the
Bulk Electric System, as defined in federal law, are subject to oversight
by the Federal Energy Regulatory Commission (FERC), the North American
Electric Reliability Corporation (NERC), and Texas Reliability Entity, Inc.
(Texas RE).
http://www.ercot.com/mktrules/compliance

ERCOT is subject to (federal) NERC Reliability Standards, but not
interstate transmission regulations.
Only generation and retail electric providers are deregulated. Transmission
and distribution are not. Municipally owned utilities and electric coop in
ERCOT region are exempt from unbundling (from vertically integrated
monopoly).

Re: Texas internet connectivity declining due to blackouts

[Rod Beck]

It will happen because storm frequency and outages will rise throughout the 
century. It is a capital investment that will sharply reduce outages. Florida 
is on the verge of putting their long-haul power underground. Texas is primed 
for storms due to the high and growing humidity in the Gulf. Finally, capital 
costs - interest - is low. Both governments and big corporates pay very little 
to borrow so it is not onerous. May be it is time to make America great again. 

Nor does it have to be mandatory for all systems. But there should be at least 
one power network in the ground as well as one telco network.

Regards,

Roderick.


From: Jared Mauch 
Sent: Tuesday, February 16, 2021 3:38 PM
To: Mike Hammett 
Cc: Rod Beck ; nanog@nanog.org 

Subject: Re: Texas internet connectivity declining due to blackouts



> On Feb 16, 2021, at 8:25 AM, Mike Hammett  wrote:
>
> It's cheaper to build 2x, 3x, 4x the aerial plant than to build 1x the 
> underground plant.
>
> The actual cost per foot is more like 10x difference, but there are right of 
> way, maintenance, etc. costs to factor in as well.
>

Labor is something in 8x but the permit/engineering cost is usually the same 
per foot, but the make-ready on poles can make underground competitive or more 
like 1.5x when you fully bake the costs.  Really depends on pole distances and 
quality.  O long-term is lower on underground vs aerial.

- Jared

Re: Texas internet connectivity declining due to blackouts

[Jared Mauch]

> On Feb 16, 2021, at 8:25 AM, Mike Hammett  wrote:
> 
> It's cheaper to build 2x, 3x, 4x the aerial plant than to build 1x the 
> underground plant.
> 
> The actual cost per foot is more like 10x difference, but there are right of 
> way, maintenance, etc. costs to factor in as well.
> 

Labor is something in 8x but the permit/engineering cost is usually the same 
per foot, but the make-ready on poles can make underground competitive or more 
like 1.5x when you fully bake the costs.  Really depends on pole distances and 
quality.  O long-term is lower on underground vs aerial.

- Jared

Re: Netflix Contact

[Cassell, Brandon]

Netflix Contact:
If that was you that caused that ticket update to happen, you have my deepest 
thanks.

Thanks,

Brandon Cassell
bcass...@oar.net


From: NANOG  on behalf of "Cassell, 
Brandon" 
Date: Monday, February 15, 2021 at 9:40 AM
To: "nanog@nanog.org" 
Subject: Netflix Contact

If anyone from Netflix is around, I’d appreciate it if you could hit me up off 
list, we have a ticket open that I could use some assistance on.

Thanks,

Brandon Cassell
bcass...@oar.net

Re: Texas internet connectivity declining due to blackouts

[Bret Clark]

Texas doesn't generally experience this type of extreme cold. The power grids 
are being overload due to people using their electric heat or electric portable 
heaters.


From: NANOG  on behalf of 
Rod Beck 
Sent: Tuesday, February 16, 2021 7:09 AM
To: Robert Jacobs ; Mark Tinka ; 
nanog@nanog.org ; Cory Sell 
Subject: Re: Texas internet connectivity declining due to blackouts

The problems with renewables is that you can't switch on or off and there is no 
good storage solution.

However, the issue in Texas is probably exposed power cables. In Europe they 
are buried and we have far milder weather than the States.

Anyone wants to provide some details on where the system has faltered? It is 
transmission? Or generation? Or just everything in general? 


From: NANOG  on behalf 
of Cory Sell via NANOG 
Sent: Tuesday, February 16, 2021 5:34 AM
To: Robert Jacobs ; Mark Tinka ; 
nanog@nanog.org 
Subject: RE: Texas internet connectivity declining due to blackouts

Ercot has already released actual documentation of the outputs. Wind is NOT the 
biggest loss here. Even if wind was operating at 100% capacity, we’d be in the 
same boat due to gas and fossil fuel-related generation being decimated. 
Estimated 4GW lost for wind doesn’t make up for the 30GW+ estimated being lost 
from fossil fuels.

I only interject because people are already pointing their fingers at 
renewables being the cause here and trying to pawn off the blame to wind/solar 
to further their agendas to reduce renewable energy R and adoption. Sure, 
wind isn’t perfect, but looks like solution relied on failed in a massive way.

Sent from ProtonMail Mobile


On Mon, Feb 15, 2021 at 10:17 PM, Robert Jacobs 
mailto:rjac...@pslightwave.com>> wrote:
How about letting us Texans have more natural gas power plants or even let the 
gas be delivered to the plants we have so they can provide more power in an 
emergency. Did not help that 20% of our power is now wind which of course in an 
ice storm like we are having is shut off... Lots of issues and plenty of 
politics involved here..

Robert Jacobs​
 |  Data Center Manager
[http://www.pslightwave.com/emailsig/plwlogo.jpg]
Direct: 832-615-7742
Mobile: 281-830-2092
Main:   832‑615‑8000
Fax:713-510-1650
5959 Corporate Dr. Suite 3300; Houston, TX 77036
[Facebook]
[LinkedIn]
[Twitter]
[http://www.pslightwave.com/emailsig/2020TopWorkplace.png]
 [http://www.pslightwave.com/emailsig/plw-wbenc.jpg] A Certified 
Woman‑Owned Business
24x7x365 Customer Support: 832-615-8000 | supp...@pslightwave.com

​This electronic message contains information from PS Lightwave which may be 
privileged and confidential. The information is intended to be for the use of 
individual(s) or entity named above. If you are not the intended recipient, any 
disclosure, copying, distribution or use of the contents of this information is 
prohibited. If you have received this electronic message in error, please 
notify me by telephone or e-mail immediately.
-Original Message-
From: NANOG  On Behalf Of Mark 
Tinka
Sent: Monday, February 15, 2021 10:06 PM
To: nanog@nanog.org
Subject: Re: Texas internet connectivity declining due to blackouts



On 2/16/21 04:14, Sean Donelan wrote:
>
> Poweroutage.us posted a terrific map, showing the jurisdictional
> borders of the Texas power outages versus the storm related power
> outages elsewhere in the country.
>
> https://twitter.com/PowerOutage_us/status/1361493394070118402
>
>
> Sometimes infrastructure planning failures are not due to "natural
> hazards."

I suppose having some kind of home backup solution wouldn't be too bad right 
now, even though you may still not get access to services. But at least, you 
can brew some coffee, and charge your pulse oximetre.

Mark.

Re: Infomart Dallas is on generator

[Keith Stokes]

Equinix DA-2 reported loads transferred 3-4 a.m.




From: NANOG  on behalf of Robert 
DeVita 
Sent: Monday, February 15, 2021 4:51 PM
To: Eric Kuhnke ; nanog@nanog.org list 
Subject: Re: Infomart Dallas is on generator

Hopefully the other 400mw in Dallas follow their lead.

Robert DeVita
Founder & CEO
Mejeticks
c. 469-441-8864
e. radev...@mejeticks.com

From: NANOG  on behalf of Eric 
Kuhnke 
Sent: Monday, February 15, 2021 4:10:32 PM
To: nanog@nanog.org list 
Subject: Infomart Dallas is on generator

I have now heard from two reliable sources that Infomart Dallas is presently on 
generator, and is likely to remain so until the cold weather/electrical supply 
emergency in Texas has abated. No network impact seen yet.

Re: Texas internet connectivity declining due to blackouts

[Michael Thomas]

On 2/16/21 3:05 AM, Jared Mauch wrote:
Almost exactly 4 years ago we were out up here in Michigan for over 
120 hours after a wind storm took out power to 1 million homes. Large 
scale restoration takes time. When the load and supply are imbalanced 
it can make things worse as well.


I'm hoping things return to normal soon but also am reminded it can 
take some time.


We now have a large generator with automatic switchover after that 
event. Filling gas cans every 12 hours to feed the generator is no fun.


We use propane. It's less dense energy-wise than gasoline, but it's 
really easy to switch over.


Mike

Re: Texas internet connectivity declining due to blackouts

[Robert DeVita]

What’s going on in Texas has nothing to do with power distribution. It has to 
do with ability to generate power.

Robert DeVita
Founder & CEO
Mejeticks
c. 469-441-8864
e. radev...@mejeticks.com

From: NANOG  on behalf of Mike 
Hammett 
Sent: Tuesday, February 16, 2021 7:25:12 AM
To: Rod Beck 
Cc: nanog@nanog.org 
Subject: Re: Texas internet connectivity declining due to blackouts

It's cheaper to build 2x, 3x, 4x the aerial plant than to build 1x the 
underground plant.

The actual cost per foot is more like 10x difference, but there are right of 
way, maintenance, etc. costs to factor in as well.



-
Mike Hammett
Intelligent Computing Solutions 

[http://www.ics-il.com/images/fbicon.png] 

 [http://www.ics-il.com/images/googleicon.png]  

 [http://www.ics-il.com/images/linkedinicon.png]  

 [http://www.ics-il.com/images/twittericon.png]  

Midwest Internet Exchange 

[http://www.ics-il.com/images/fbicon.png] 

 [http://www.ics-il.com/images/linkedinicon.png]  

 [http://www.ics-il.com/images/twittericon.png]  

The Brothers WISP 

[http://www.ics-il.com/images/fbicon.png] 

 [http://www.ics-il.com/images/youtubeicon.png]  


From: "Rod Beck" 
To: "Sean Donelan" , "Mikael Abrahamsson" 
Cc: nanog@nanog.org
Sent: Tuesday, February 16, 2021 6:05:41 AM
Subject: Re: Texas internet connectivity declining due to blackouts

Are the power lines buried like in Europe where I live?

I really think using poles is crazy and global warming guarantees enough 
atmospheric turbulence to make it untenable. Florida is moving to bury power 
lines.


From: NANOG  on behalf 
of Mikael Abrahamsson via NANOG 
Sent: Tuesday, February 16, 2021 9:06 AM
To: Sean Donelan 
Cc: nanog@nanog.org 
Subject: RE: Texas internet connectivity declining due to blackouts
On Mon, 15 Feb 2021, Sean Donelan wrote:

> Strange the massive shortages and failures are only in one state.
>
> The extreme cold weather extends northwards across many states, which aren't
> reporting rolling blackouts.

https://www.texastribune.org/2011/02/08/texplainer-why-does-texas-have-its-own-power-grid/
 


Going at it alone can be beneficial sometimes, sometimes it's not.

--
Mikael Abrahamssonemail: swm...@swm.pp.se

Re: Texas internet connectivity declining due to blackouts

[Mike Hammett]

It's cheaper to build 2x, 3x, 4x the aerial plant than to build 1x the 
underground plant. 


The actual cost per foot is more like 10x difference, but there are right of 
way, maintenance, etc. costs to factor in as well. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Rod Beck"  
To: "Sean Donelan" , "Mikael Abrahamsson"  
Cc: nanog@nanog.org 
Sent: Tuesday, February 16, 2021 6:05:41 AM 
Subject: Re: Texas internet connectivity declining due to blackouts 


Are the power lines buried like in Europe where I live? 



I really think using poles is crazy and global warming guarantees enough 
atmospheric turbulence to make it untenable. Florida is moving to bury power 
lines. 






From: NANOG  on behalf 
of Mikael Abrahamsson via NANOG  
Sent: Tuesday, February 16, 2021 9:06 AM 
To: Sean Donelan  
Cc: nanog@nanog.org  
Subject: RE: Texas internet connectivity declining due to blackouts 


On Mon, 15 Feb 2021, Sean Donelan wrote: 

> Strange the massive shortages and failures are only in one state. 
> 
> The extreme cold weather extends northwards across many states, which aren't 
> reporting rolling blackouts. 

https://www.texastribune.org/2011/02/08/texplainer-why-does-texas-have-its-own-power-grid/
 

Going at it alone can be beneficial sometimes, sometimes it's not. 

-- 
Mikael Abrahamsson email: swm...@swm.pp.se 

Re: Texas internet connectivity declining due to blackouts

[Sean Donelan]

On Tue, 16 Feb 2021, Rod Beck wrote:

Are the power lines buried like in Europe where I live?


Rolling blackouts in Texas (or elsewhere) are not caused by storm damage.

Rolling blackouts are administrative actions (turn off power, turn on 
power) taken by the system operator.  They can "turn on" the power after a 
rolling blackout with a switch because there is no damage to the outside 
plant.


In California, they use rolling blackouts BEFORE wildfires to prevent 
power line sparking causing wildfires. Not because of damage to the 
outside plant. In Texas, they use rolling blackouts because they didn't 
have enough generation capacity online.


Again, the rolling blackouts in Texas is not due to storm damage.  Its 
because natural gas power generator plants froze, and Texas is an 
separate electric grid from the rest of North America due to political 
reasons. That's why the rolling blackouts stop at the Texas border 
(approximately).


Chicago Illinois (3rd largest metro area) power plants haven't frozen, and 
can use power from states across the north-central US grid.


Most of Texas is rural and cheap. Power lines are built above ground. 
Only in a few dense urban areas, i.e. downtown Dallas or downtown 
Houston, are power lines underground for aesthetic reasons.

Re: Texas internet connectivity declining due to blackouts

[John Sage]

On 2/16/21 4:22 AM, John Sage wrote:

On 2/15/21 10:02 PM, Mark Tinka wrote:



On 2/16/21 07:49, Matthew Petach wrote:



Isn't that a result of ERCOT stubbornly refusing to interconnect with 
the rest of the national grid, out of an irrational fear of coming 
under federal regulation?


Yes. This has been widely documented in numerous articles, both very 
recently and previously.




As one example only, of many:

"What went wrong with the Texas power grid?"

Marcy de Luna, Amanda Drane, Houston Chronicle

Feb. 15, 2021
Updated: Feb. 15, 2021 9:23 p.m.

"Dan Woodfin, ERCOT’s senior director of system operations, said the 
rolling blackouts are taking more power offline for longer periods than 
ever before. An estimated 34,000 megawatts of power generation — more 
than a third of the system’s total generating capacity — had been 
knocked offline by the extreme winter weather amid soaring demand as 
residents crank up heating systems."


. . .

"Ed Hirs, an energy fellow in the Department of Economics at the 
University of Houston, blamed the failures on the state’s deregulated 
power system, which doesn’t provide power generators with the returns 
needed to invest in maintaining and improving power plants.


“The ERCOT grid has collapsed in exactly the same manner as the old 
Soviet Union,” said Hirs. “It limped along on underinvestment and 
neglect until it finally broke under predictable circumstances."


https://www.houstonchronicle.com/business/energy/article/Wholesale-power-prices-spiking-across-Texas-15951684.php 




- John
__

Re: Texas internet connectivity declining due to blackouts

[John Sage]

On 2/15/21 10:02 PM, Mark Tinka wrote:



On 2/16/21 07:49, Matthew Petach wrote:



Isn't that a result of ERCOT stubbornly refusing to interconnect with 
the rest of the national grid, out of an irrational fear of coming 
under federal regulation?


Yes. This has been widely documented in numerous articles, both very 
recently and previously.



I suspect that trying to be self-sufficient works most of the 
time--but when you get to the edges of the bell curve locally, your 
ability to be resilient and survive depends heavily upon your ability 
to be supported by others around you.  This certainly holds true for 
individual humans; I suspect power grids aren't that different.


If there was a state-wide blackout, they'd need to restart from the 
national grid anyway. Why not have some standing interconnection 
agreement with them anyway, that gets activated in cases such as these?


Sorry, unfamiliar with U.S. politics in this regard, so just doing 1+1.



"Sorry, unfamiliar with U.S. politics in this regard, so just doing 1+1"

You don't understand Texas politics relative to the United States at large.

Which is fine, but this is a state that had deliberately prevented 
interconnects (see: ERCOT, above) into any extended national grid, 
principally to evade the resulting exposure to Federal regulation.


Texas [politicians] are constantly threatening to secede.


- John
--

Re: Texas internet connectivity declining due to blackouts

[Rod Beck]

I agree. Germany spent well over 200 billion Euros on wind and solar subsidies 
and over 85% of the country's energy consumption is still non-renewable. Wind 
power is randomly generated. I really don't to depend it for either personal or 
business needs.


From: NANOG  on behalf 
of Robert Jacobs 
Sent: Tuesday, February 16, 2021 5:17 AM
To: Mark Tinka ; nanog@nanog.org 
Subject: RE: Texas internet connectivity declining due to blackouts

How about letting us Texans have more natural gas power plants or even let the 
gas be delivered to the plants we have so they can provide more power in an 
emergency. Did not help that 20% of our power is now wind which of course in an 
ice storm like we are having is shut off... Lots of issues and plenty of 
politics involved here..

Robert Jacobs​
 |  Data Center Manager
[http://www.pslightwave.com/emailsig/plwlogo.jpg]
Direct: 832-615-7742
Mobile: 281-830-2092
Main:   832‑615‑8000
Fax:713-510-1650
5959 Corporate Dr. Suite 3300; Houston, TX 77036
[Facebook]
[LinkedIn]
[Twitter]
[http://www.pslightwave.com/emailsig/2020TopWorkplace.png]
 [http://www.pslightwave.com/emailsig/plw-wbenc.jpg] A Certified 
Woman‑Owned Business
24x7x365 Customer Support: 832-615-8000 | supp...@pslightwave.com

​This electronic message contains information from PS Lightwave which may be 
privileged and confidential. The information is intended to be for the use of 
individual(s) or entity named above. If you are not the intended recipient, any 
disclosure, copying, distribution or use of the contents of this information is 
prohibited. If you have received this electronic message in error, please 
notify me by telephone or e-mail immediately.
-Original Message-
From: NANOG  On Behalf Of Mark 
Tinka
Sent: Monday, February 15, 2021 10:06 PM
To: nanog@nanog.org
Subject: Re: Texas internet connectivity declining due to blackouts



On 2/16/21 04:14, Sean Donelan wrote:
>
> Poweroutage.us posted a terrific map, showing the jurisdictional
> borders of the Texas power outages versus the storm related power
> outages elsewhere in the country.
>
> https://twitter.com/PowerOutage_us/status/1361493394070118402
>
>
> Sometimes infrastructure planning failures are not due to "natural
> hazards."

I suppose having some kind of home backup solution wouldn't be too bad right 
now, even though you may still not get access to services. But at least, you 
can brew some coffee, and charge your pulse oximetre.

Mark.

Re: Texas internet connectivity declining due to blackouts

[Rod Beck]

The problems with renewables is that you can't switch on or off and there is no 
good storage solution.

However, the issue in Texas is probably exposed power cables. In Europe they 
are buried and we have far milder weather than the States.

Anyone wants to provide some details on where the system has faltered? It is 
transmission? Or generation? Or just everything in general? 


From: NANOG  on behalf 
of Cory Sell via NANOG 
Sent: Tuesday, February 16, 2021 5:34 AM
To: Robert Jacobs ; Mark Tinka ; 
nanog@nanog.org 
Subject: RE: Texas internet connectivity declining due to blackouts

Ercot has already released actual documentation of the outputs. Wind is NOT the 
biggest loss here. Even if wind was operating at 100% capacity, we’d be in the 
same boat due to gas and fossil fuel-related generation being decimated. 
Estimated 4GW lost for wind doesn’t make up for the 30GW+ estimated being lost 
from fossil fuels.

I only interject because people are already pointing their fingers at 
renewables being the cause here and trying to pawn off the blame to wind/solar 
to further their agendas to reduce renewable energy R and adoption. Sure, 
wind isn’t perfect, but looks like solution relied on failed in a massive way.

Sent from ProtonMail Mobile


On Mon, Feb 15, 2021 at 10:17 PM, Robert Jacobs 
mailto:rjac...@pslightwave.com>> wrote:
How about letting us Texans have more natural gas power plants or even let the 
gas be delivered to the plants we have so they can provide more power in an 
emergency. Did not help that 20% of our power is now wind which of course in an 
ice storm like we are having is shut off... Lots of issues and plenty of 
politics involved here..

Robert Jacobs​
 |  Data Center Manager
[http://www.pslightwave.com/emailsig/plwlogo.jpg]
Direct: 832-615-7742
Mobile: 281-830-2092
Main:   832‑615‑8000
Fax:713-510-1650
5959 Corporate Dr. Suite 3300; Houston, TX 77036
[Facebook]
[LinkedIn]
[Twitter]
[http://www.pslightwave.com/emailsig/2020TopWorkplace.png]
 [http://www.pslightwave.com/emailsig/plw-wbenc.jpg] A Certified 
Woman‑Owned Business
24x7x365 Customer Support: 832-615-8000 | supp...@pslightwave.com

​This electronic message contains information from PS Lightwave which may be 
privileged and confidential. The information is intended to be for the use of 
individual(s) or entity named above. If you are not the intended recipient, any 
disclosure, copying, distribution or use of the contents of this information is 
prohibited. If you have received this electronic message in error, please 
notify me by telephone or e-mail immediately.
-Original Message-
From: NANOG  On Behalf Of Mark 
Tinka
Sent: Monday, February 15, 2021 10:06 PM
To: nanog@nanog.org
Subject: Re: Texas internet connectivity declining due to blackouts



On 2/16/21 04:14, Sean Donelan wrote:
>
> Poweroutage.us posted a terrific map, showing the jurisdictional
> borders of the Texas power outages versus the storm related power
> outages elsewhere in the country.
>
> https://twitter.com/PowerOutage_us/status/1361493394070118402
>
>
> Sometimes infrastructure planning failures are not due to "natural
> hazards."

I suppose having some kind of home backup solution wouldn't be too bad right 
now, even though you may still not get access to services. But at least, you 
can brew some coffee, and charge your pulse oximetre.

Mark.

Re: Texas internet connectivity declining due to blackouts

[Rod Beck]

Are the power lines buried like in Europe where I live?

I really think using poles is crazy and global warming guarantees enough 
atmospheric turbulence to make it untenable. Florida is moving to bury power 
lines.


From: NANOG  on behalf 
of Mikael Abrahamsson via NANOG 
Sent: Tuesday, February 16, 2021 9:06 AM
To: Sean Donelan 
Cc: nanog@nanog.org 
Subject: RE: Texas internet connectivity declining due to blackouts

On Mon, 15 Feb 2021, Sean Donelan wrote:

> Strange the massive shortages and failures are only in one state.
>
> The extreme cold weather extends northwards across many states, which aren't
> reporting rolling blackouts.

https://www.texastribune.org/2011/02/08/texplainer-why-does-texas-have-its-own-power-grid/

Going at it alone can be beneficial sometimes, sometimes it's not.

--
Mikael Abrahamssonemail: swm...@swm.pp.se

Re: Texas internet connectivity declining due to blackouts

[Jared Mauch]

Almost exactly 4 years ago we were out up here in Michigan for over 120 hours 
after a wind storm took out power to 1 million homes. Large scale restoration 
takes time. When the load and supply are imbalanced it can make things worse as 
well. 

I'm hoping things return to normal soon but also am reminded it can take some 
time. 

We now have a large generator with automatic switchover after that event. 
Filling gas cans every 12 hours to feed the generator is no fun. 

Sent from my TI-99/4a

> On Feb 15, 2021, at 11:54 PM, Cory Sell via NANOG  wrote:
> 
>  Total population is a pretty big difference as you go north, as is how well 
> infrastructure is actually prepared for snow/ice and cold temperatures in 
> general.
> 
> I’ve been without power all day and have no doubt I’ll cross the 24-hour mark 
> here in a handful of hours.
> 
> Sent from ProtonMail Mobile
> 
> 
>> On Mon, Feb 15, 2021 at 10:42 PM, Sean Donelan  wrote:
>> 
>> 
>> On Tue, 16 Feb 2021, Cory Sell via NANOG wrote:
>> > adoption. Sure, wind isn’t perfect, but looks like solution relied on 
>> > failed
>> > in a massive way.
>> 
>> Strange the massive shortages and failures are only in one state.
>> 
>> The extreme cold weather extends northwards across many states, which
>> aren't reporting rolling blackouts.
> 
> 

RE: Texas internet connectivity declining due to blackouts

[Mikael Abrahamsson via NANOG]

On Mon, 15 Feb 2021, Sean Donelan wrote:


Strange the massive shortages and failures are only in one state.

The extreme cold weather extends northwards across many states, which aren't 
reporting rolling blackouts.


https://www.texastribune.org/2011/02/08/texplainer-why-does-texas-have-its-own-power-grid/

Going at it alone can be beneficial sometimes, sometimes it's not.

--
Mikael Abrahamssonemail: swm...@swm.pp.se