Re: Power cut if temps are too high

[Owen DeLong]

> On May 28, 2019, at 21:27 , b...@theworld.com  wrote:
> 
> 
> On May 28, 2019 at 19:56 o...@delong.com  (Owen 
> DeLong) wrote:
>> It’s unlikely to apply to much of anything in a datacenter other than disks.
> 
> Ok, disks, a mere bagatelle of a concern.
> 
> Then again obviously disks have gotten much, much better about thermal
> change since people in, e.g., temperate climates might take their
> laptops' running disks from a long, frigid walk into a warm building.
> 
> There was a time when you weren't supposed to move a disk while it was
> still spinning (e.g., hot swaps had to be able to cut power before
> removal so you could give the heads several seconds to stop and park),
> not sure how they solved that so completely, again, laptops.

A big part of solving this was smaller platters. Remember, platters from the 
time you’re talking about were somewhere between 5.25” diameter and 14” or even 
19” diameter in some cases. Platters were usually made of fairly rigid steel 
and the heads were separated from the platters by air cushions and Bernouli’s 
principal and very very little else.

Today’s disk drives have the heads much closer to the media, but the media is a 
lot less sensitive to brief head contact. The media is often on thin flexible 
plastic substrates and the separation of the heads is often purely mechanical.

Head insertion and parking mechanisms have changed quite a bit as well, 
allowing for much sturdier gantries (the head assembly is now usually mounted 
on a triangular gantry with an arced side where the actuators connect. This 
allows a significantly larger amount of material to be used in constructing the 
gantry and the heads travel in an arc across the media instead of in a linear 
motion as was common in older drives.

> 
>> The reason it applies to disks is because rapid cooling of a drive will lead 
>> to uneven cooling of the platters which may cause abnormal stresses leading 
>> to shattering and/or warpage (depending on the material the drive platters 
>> are made from).
> 
> Also head clearances and other moving parts tolerances.

Well, the primary thing that’s going to cause you grief for abrupt temperature 
changes is when the platters warp, resulting in reduced head clearance 
(possibly even negative head clearance). In the case of some glass platters, 
shattering will also make for a really bad day.

> 
>> 
>> Most electronic components can tolerate a pretty steep thermal curve in 
>> either direction so long as the curve doesn’t take them out of spec one way 
>> or the other.
>> 
>> Also, most circuit boards and the like do not have enough mass to surface 
>> area ratio to lead to significant temperature differentials within a small 
>> physical distance.
> 
> Then again if you're cooling a room from, say, 115F to 70F you only
> need one excuse to consider the rate of cooling and disks would be a
> pretty good excuse.
> 
> SSDs no doubt are obsoleting even that concern.
> 
> But I still tend to worry about the relationship of resistance to
> temperature in circuits as a general principle tho perhaps in the
> likely range it's not a major concern.

Modern resistors don’t tend to move as much as in the past. However, even for 
that, as long as you don’t exceed the operating temperature thresholds at 
either end, you should be fine. The rate of cooling/heating isn’t really an 
issue for that.

> Anyhow, IT'S WORTH A THOUGHT if something extreme happens to
> temperatures in your machine room.

If you get out of tolerance, then there are lots of things to think about. If 
you’re within allowed operating range for your equipment, lots less.

Rate of change, as I said, is the one that’s unique to disk drives due to the 
high mass to surface area ratio and the tight mechanical tolerances.


> You might not want to fling open the doors and windows of a 110+F room
> to 0F outside air and begin turning everything back on as the room's
> air thermometer begins to register 70F a few minutes later.

Yep… If you’ve got spinning media, that’s probably not the best plan. For solid 
state stuff, it’s probably OK as long as you don’t leave the door open until 
the room starts to drop below 10ºC.

> Water condensation can also be a concern, after a prolonged A/C
> failure it may be hot and humid in the room depending on the climate
> etc.

Truth. If you create a new weather pattern inside the datacenter, you’re 
conducting some form of experiment likely to yield “interesting” results.

Owen

> 
> File Under: MORE THINGS TO WORRY ABOUT!
> 
>> Owen
>> 
>> 
>>> On May 28, 2019, at 12:18 , b...@theworld.com wrote:
>>> 
>>> 
>>> Something to keep in mind is that some equipment, disks in particular,
>>> should only be cooled at a certain rate once they're hot, often
>>> annoyingly slow by the specs like 2-3 degrees C per hour but there are
>>> probably circuits sensitive to this also which could be anywhere.
>>> 
>>> It came up because it happened to me in Cambridge, 

Re: Power cut if temps are too high

[bzs]

On May 28, 2019 at 19:56 o...@delong.com (Owen DeLong) wrote:
 > It’s unlikely to apply to much of anything in a datacenter other than disks.

Ok, disks, a mere bagatelle of a concern.

Then again obviously disks have gotten much, much better about thermal
change since people in, e.g., temperate climates might take their
laptops' running disks from a long, frigid walk into a warm building.

There was a time when you weren't supposed to move a disk while it was
still spinning (e.g., hot swaps had to be able to cut power before
removal so you could give the heads several seconds to stop and park),
not sure how they solved that so completely, again, laptops.

 > The reason it applies to disks is because rapid cooling of a drive will lead 
 > to uneven cooling of the platters which may cause abnormal stresses leading 
 > to shattering and/or warpage (depending on the material the drive platters 
 > are made from).

Also head clearances and other moving parts tolerances.

 > 
 > Most electronic components can tolerate a pretty steep thermal curve in 
 > either direction so long as the curve doesn’t take them out of spec one way 
 > or the other.
 > 
 > Also, most circuit boards and the like do not have enough mass to surface 
 > area ratio to lead to significant temperature differentials within a small 
 > physical distance.

Then again if you're cooling a room from, say, 115F to 70F you only
need one excuse to consider the rate of cooling and disks would be a
pretty good excuse.

SSDs no doubt are obsoleting even that concern.

But I still tend to worry about the relationship of resistance to
temperature in circuits as a general principle tho perhaps in the
likely range it's not a major concern.

Anyhow, IT'S WORTH A THOUGHT if something extreme happens to
temperatures in your machine room.

You might not want to fling open the doors and windows of a 110+F room
to 0F outside air and begin turning everything back on as the room's
air thermometer begins to register 70F a few minutes later.

Water condensation can also be a concern, after a prolonged A/C
failure it may be hot and humid in the room depending on the climate
etc.

File Under: MORE THINGS TO WORRY ABOUT!

 > Owen
 > 
 > 
 > > On May 28, 2019, at 12:18 , b...@theworld.com wrote:
 > > 
 > > 
 > > Something to keep in mind is that some equipment, disks in particular,
 > > should only be cooled at a certain rate once they're hot, often
 > > annoyingly slow by the specs like 2-3 degrees C per hour but there are
 > > probably circuits sensitive to this also which could be anywhere.
 > > 
 > > It came up because it happened to me in Cambridge, MA in the dead of
 > > winter and every helpful person in the building came by to suggest I
 > > just open windows and doors to the snowy outdoors to get things
 > > running sooner.
 > > 
 > > It should be in the specs and if you're concerned about equipment
 > > running in too hot an environment you might be concerned about this
 > > also. Particularly after a forced power-down which also powers down
 > > equipment fans while the chips etc are still hot so will continue
 > > heating cases.
 > > 
 > > Ambient air temperature might not be telling you the whole story is
 > > the point.
 > > 
 > > I keep one of those big 5' fans, looks like something they use in
 > > Hollywood for windstorms and feels a bit like it on high, for just
 > > this sort of reason tho even if I just think it's getting warm, and
 > > several smaller fans to point at racks etc.
 > > 
 > > The best thing you can do if it gets too hot is keep the air moving.
 > > 
 > > (Where to plug the fans in after a power shutdown is your problem, I
 > > knew someone would think that!)
 > > 
 > > -- 
 > >-Barry Shein
 > > 
 > > Software Tool & Die| b...@theworld.com | 
 > > http://www.TheWorld.com
 > > Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
 > > The World: Since 1989  | A Public Information Utility | *oo*
 > 

-- 
-Barry Shein

Software Tool & Die| b...@theworld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
The World: Since 1989  | A Public Information Utility | *oo*

Re: Power cut if temps are too high

[Owen DeLong]

It’s unlikely to apply to much of anything in a datacenter other than disks.

The reason it applies to disks is because rapid cooling of a drive will lead to 
uneven cooling of the platters which may cause abnormal stresses leading to 
shattering and/or warpage (depending on the material the drive platters are 
made from).

Most electronic components can tolerate a pretty steep thermal curve in either 
direction so long as the curve doesn’t take them out of spec one way or the 
other.

Also, most circuit boards and the like do not have enough mass to surface area 
ratio to lead to significant temperature differentials within a small physical 
distance.

Owen


> On May 28, 2019, at 12:18 , b...@theworld.com wrote:
> 
> 
> Something to keep in mind is that some equipment, disks in particular,
> should only be cooled at a certain rate once they're hot, often
> annoyingly slow by the specs like 2-3 degrees C per hour but there are
> probably circuits sensitive to this also which could be anywhere.
> 
> It came up because it happened to me in Cambridge, MA in the dead of
> winter and every helpful person in the building came by to suggest I
> just open windows and doors to the snowy outdoors to get things
> running sooner.
> 
> It should be in the specs and if you're concerned about equipment
> running in too hot an environment you might be concerned about this
> also. Particularly after a forced power-down which also powers down
> equipment fans while the chips etc are still hot so will continue
> heating cases.
> 
> Ambient air temperature might not be telling you the whole story is
> the point.
> 
> I keep one of those big 5' fans, looks like something they use in
> Hollywood for windstorms and feels a bit like it on high, for just
> this sort of reason tho even if I just think it's getting warm, and
> several smaller fans to point at racks etc.
> 
> The best thing you can do if it gets too hot is keep the air moving.
> 
> (Where to plug the fans in after a power shutdown is your problem, I
> knew someone would think that!)
> 
> -- 
>-Barry Shein
> 
> Software Tool & Die| b...@theworld.com | 
> http://www.TheWorld.com
> Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
> The World: Since 1989  | A Public Information Utility | *oo*

Re: Power cut if temps are too high

[bzs]

Something to keep in mind is that some equipment, disks in particular,
should only be cooled at a certain rate once they're hot, often
annoyingly slow by the specs like 2-3 degrees C per hour but there are
probably circuits sensitive to this also which could be anywhere.

It came up because it happened to me in Cambridge, MA in the dead of
winter and every helpful person in the building came by to suggest I
just open windows and doors to the snowy outdoors to get things
running sooner.

It should be in the specs and if you're concerned about equipment
running in too hot an environment you might be concerned about this
also. Particularly after a forced power-down which also powers down
equipment fans while the chips etc are still hot so will continue
heating cases.

Ambient air temperature might not be telling you the whole story is
the point.

I keep one of those big 5' fans, looks like something they use in
Hollywood for windstorms and feels a bit like it on high, for just
this sort of reason tho even if I just think it's getting warm, and
several smaller fans to point at racks etc.

The best thing you can do if it gets too hot is keep the air moving.

(Where to plug the fans in after a power shutdown is your problem, I
knew someone would think that!)

-- 
-Barry Shein

Software Tool & Die| b...@theworld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
The World: Since 1989  | A Public Information Utility | *oo*

Anyone from OpenDNS

[sam]

Good afternoon list,
Anyone on the list from OpenDNS willing to contact me offlist? Somehow 
my $dayjobs SSL cert if being munged on your service.


Thank you
Sam Moats

Re: Flexible OTN / fractional 100GbE

[Jérôme Nicolle]

Hello Jason,

Thanks for your answer.

Le 28/05/2019 à 13:02, Jason Lixfeld a écrit :
> IP Infusion’s OcNOS is geared towards OCP gear, and while it’s not
> exactly what you’re looking for, 

I didn't see any mention of OTN in their software's specs.

OTN is important for this project because we'd need many of its features
in terms of FEC, monitoring and low jitter.

> they recently published[1] a note
> pertaining to IPoDWDM, so I could see them having maybe already done
> something along the lines of what you’re asking about, or they may
> have plans to.

We're not really working on the optical side of things, it's really just
about replacing Ethernet wherever it's relevant. Regionnal to long-haul
P2P links for that matter.

Best regards,

-- 
Jérôme Nicolle
+33 6 19 31 27 14

Re: Power cut if temps are too high

[Landon Stewart]

On May 27, 2019, at 11:00 AM, Dovid Bender  wrote:
> 
> Hi,
> 
> Is anyone aware of a device that will cut the power if the room goes above X 
> degrees? I am looking for something as a just in case.

I would personally make one with an ESP32 (micro controller with wifi) and a 
(normally closed) relay.  You could trigger it to open the circuit when the 
temperature is too high.

Did a quick google for this and found something to at least explain wtf I’m 
talking about:
https://techtutorialsx.com/2018/02/17/esp32-arduino-controlling-a-relay/ 


--
Landon Stewart
Lead Analyst - Abuse and Security Management
INAP®
📧 lstew...@inap.com
🌍 www.inap.com



signature.asc
Description: Message signed with OpenPGP

Re: Power cut if temps are too high

[Nick Hilliard]

Warren Kumari wrote on 28/05/2019 14:45:

There isn't much point to this story, but I've got a cold, and wanted
to share...:-P


whoever brought this cold to the RIPE meeting and infected a bunch of us 
has a lot to answer for, damn them. :-(


Nick

Re: Power cut if temps are too high

[Warren Kumari]

I used to work for a small, fairly crappy ISP -- the "datacenter" was
a converted brick garage / loading dock. In order to provide cooling,
they had chipped out a bunch of bricks, and mounted in 8 or so AC
units, all in a line.

We monitored everything with WhatsUp Gold[0] - one (hot) night I'm
oncall, and at 3:30AM I get an alert that the environmental sensors on
one of the routers thinks it's too hot. I'm tired and grumpy, and it's
only slightly too hot, so I ack it and go back to bed. A short while
later I get paged again - another router now thinks it is
uncomfortably warm. Still grumpy, so I ack that too, and back to bed.
Sure enough, 20 minutes later, another page Fine, I get dressed,
drive over to the location -- and realize that bricks / mortar are
strong in compression, but weak in tension - the AC window units have
been quietly vibrating for many years, and the entire row of bricks
above the AC units has popped out. All the AC units are lying outside
the building on the grass, still running :-) I stared at them for
a bit, unsure what to do -- so I turned them off, bumped up the
monitoring levels, and went back to bed... Next day we blocked up the
hole, installed some temporary chillers, and then finally installed
real colling

There isn't much point to this story, but I've got a cold, and wanted
to share... :-P

W
[0]: Wow, I just realized that WUG still exists... huh.

On Tue, May 28, 2019 at 9:13 AM Thomas Bellman  wrote:
>
> On 2019-05-27 18:18 +, Mel Beckman wrote:
>
> > Before the trigger temperature is reached, the NMS would have sent
> > various escalating alarms to on call staffers, who hopefully would
> > intervene before this point.
>
> Would they actually have time to react and do something?  In our
> datacenters, we reach our cut-off temperature in about 20 minutes
> if cooling stops.
>
>
> > This system has triggered one time, successfully shutting down the data
> > center on a holiday weekend when people missed their notifications, and
> > undoubtedly saved a lot of hard drives. When we got to the room the
> > temperature was over 115°, but the power was cut at 95°.
>
> Presumably that was °F, not °C.
>
> I have heard from people who did *not* have automatic cutting of the
> power at high temperatures.  Their computer room reached 100°C in
> places; some keyboards apparently looked like a certain Salvador Dali
> painting afterwards...  (But I think they had very few actual servers
> or disk drives breaking.)  The reason it didn't get even hotter, was
> that as temperature rose, servers started overheating and shut them-
> selves down, thus lowering power disippation more and more.
>
>
> Our system for cutting power at high temperatures is part of the PLC
> monitoring power and temperature in the computer rooms.  It sends a
> signal to the large breakers connecting the power subcentrals (where
> all the 16A fuses are) to the power rail feeding the room.  I believe
> our PLCs are from Schneider Electric, but anyone who delivers PLCs
> for controlling power and cooling in a datacenter should be capable
> or programming their PLCs to do the same.  You just need to remember
> putting it in the specifications when you contract the building. :-)
>
>
> /Bellman
>


-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

Re: Power cut if temps are too high

[Thomas Bellman]

On 2019-05-27 18:18 +, Mel Beckman wrote:

> Before the trigger temperature is reached, the NMS would have sent
> various escalating alarms to on call staffers, who hopefully would
> intervene before this point.

Would they actually have time to react and do something?  In our
datacenters, we reach our cut-off temperature in about 20 minutes
if cooling stops.


> This system has triggered one time, successfully shutting down the data
> center on a holiday weekend when people missed their notifications, and
> undoubtedly saved a lot of hard drives. When we got to the room the
> temperature was over 115°, but the power was cut at 95°.

Presumably that was °F, not °C.

I have heard from people who did *not* have automatic cutting of the
power at high temperatures.  Their computer room reached 100°C in
places; some keyboards apparently looked like a certain Salvador Dali
painting afterwards...  (But I think they had very few actual servers
or disk drives breaking.)  The reason it didn't get even hotter, was
that as temperature rose, servers started overheating and shut them-
selves down, thus lowering power disippation more and more.


Our system for cutting power at high temperatures is part of the PLC
monitoring power and temperature in the computer rooms.  It sends a
signal to the large breakers connecting the power subcentrals (where
all the 16A fuses are) to the power rail feeding the room.  I believe
our PLCs are from Schneider Electric, but anyone who delivers PLCs
for controlling power and cooling in a datacenter should be capable
or programming their PLCs to do the same.  You just need to remember
putting it in the specifications when you contract the building. :-)


/Bellman



signature.asc
Description: OpenPGP digital signature

Re: Flexible OTN / fractional 100GbE

[Jason Lixfeld]

> On May 28, 2019, at 6:41 AM, Jérôme Nicolle  wrote:
> 
> Hi NaNOG !
> 
> I'm looking for a muxponder that would take OTU4s on the network side
> and provide 10/40/100GbE on the client side, with some kind of
> oversubscription, as to provide a "fractional 100GbE" e.g. starting with
> 30-60Gbps commit that could be upgraded to 100GbE when network capacity
> is available.
> 
> Is that something feasible at a decent price ?
> 
> I've read that Broadcom' StrataDNX (Qumran / Jericho) chips have OTN
> support in addition to ethernet now, is there some vendor who leverages
> this, preferably with OCP gear ?

Hi,

IP Infusion’s OcNOS is geared towards OCP gear, and while it’s not exactly what 
you’re looking for, they recently published[1] a note pertaining to IPoDWDM, so 
I could see them having maybe already done something along the lines of what 
you’re asking about, or they may have plans to.

[1] 
https://www.ipinfusion.com/news-events/ip-infusion-qualifies-inphi-colorz-in-its-latest-release-of-the-ocnos-network-operating-system/

Flexible OTN / fractional 100GbE

[Jérôme Nicolle]

Hi NaNOG !

I'm looking for a muxponder that would take OTU4s on the network side
and provide 10/40/100GbE on the client side, with some kind of
oversubscription, as to provide a "fractional 100GbE" e.g. starting with
30-60Gbps commit that could be upgraded to 100GbE when network capacity
is available.

Is that something feasible at a decent price ?

I've read that Broadcom' StrataDNX (Qumran / Jericho) chips have OTN
support in addition to ethernet now, is there some vendor who leverages
this, preferably with OCP gear ?

Thanks !

-- 
Jérôme Nicolle
+33 6 19 31 27 14

China Telecom people at NANOG 75

[Töma Gavrichenkov]

Peace,

There was some sales manager at the San Francisco meeting, from China
Telecom, whom I had a chat with.  I have lost their business card
since then, and the attendee list for the 75th meeting is already
missing from cvent dot com.

If they are reading the mailing list, then I'd ask them to drop me a
message off-list.

Have a good day.

| Töma Gavrichenkov
| gpg: 2deb 97b1 0a3c 151d b67f 1ee5 00e7 94bc 4d08 9191
| mailto: xima...@gmail.com
| fb: ximaera
| telegram: xima_era
| skype: xima_era
| tel. no: +7 916 515 49 58

Re: Power cut if temps are too high

[Wayne Bouchard]

Time Delay Relays are available with fixed or variable settings. if
you're going the mechanical approach vs scripted monitor and SNMP sort
of trigger, you can use this to cause a standard relay or SCR to trip
to raise the alarm (and hopefully also flash a warning light and/or
audibly sound an alert where people are supposed to be) when both
sensors read positive and then have the TDR do its thing when the
timer expires.

Word of caution though... any system like this needs to have some sort
of a reset and bypass in case anyone can actually catch it before it
goes down and restore environmentals rather than taking the hard
outage since that alone does lots of damage to equipment that has been
in place for a good while. You also probably ought to make sure that
the present state of said system and its pieces are visible so you can
make sure you're going to restart correctly.

-Wayne

On Mon, May 27, 2019 at 06:20:36PM +, Mel Beckman wrote:
> We considered this approach, but we wanted to have notifications precede shut 
> down, and give a remote support person the ability to prevent the shut down. 
> Our SNMP based system gives us that option.
> 
>  -mel 
> 
> > On May 27, 2019, at 11:16 AM, Brian Kantor  wrote:
> > 
> > A simple air conditioner thermostat wired to the EPO switch.
> > For safety, wire two thermostats in series so BOTH have to trip
> > before power is shut off.
> > 
> > Note that the EPO rarely does an orderly shutdown, but then this
> > is a sort of an emergency.
> >- Brian
> > 
> > 
> >> On Mon, May 27, 2019 at 02:00:39PM -0400, Dovid Bender wrote:
> >> Hi,
> >> 
> >> Is anyone aware of a device that will cut the power if the room goes above 
> >> X
> >> degrees? I am looking for something as a just in case. 
> >> 
> >> 
> >> Regards,
> >> 
> >> Dovid
> >> 

---
Wayne Bouchard
w...@typo.org
Network Dude
http://www.typo.org/~web/