Send VoiceOps mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://puck.nether.net/mailman/listinfo/voiceops
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of VoiceOps digest..."
Today's Topics:
1. Re: Sandy Watch (Darren Schreiber)
2. Re: Sandy Watch (Frank Bulk)
----------------------------------------------------------------------
Message: 1
Date: Wed, 31 Oct 2012 20:00:22 -0700
From: Darren Schreiber <[email protected]>
To: Darren Schreiber <[email protected]>, Matt Yaklin
<[email protected]>, Shripal Daphtary <[email protected]>
Cc: "[email protected]" <[email protected]>
Subject: Re: [VoiceOps] Sandy Watch
Message-ID: <[email protected]>
Content-Type: text/plain; charset="us-ascii"
Oops, I included the wrong picture link for Peer1. And no, I'm not getting
paid by them for ads or anything (we don't even host there - it's just a
cool story).
http://status.squarespace.com/ . The pictures are just too crazy.
Also, to be clear on NAC, they've been rock solid for us, the internet
blips are likely transport/fiber between us in California and NAC in New
Jersey.
Hope everyone weathered the storm OK. Good luck out there.
On 10/31/12 7:56 PM, "Darren Schreiber" <[email protected]> wrote:
>I also saw this interesting article on 911 emergency service availability.
>It was an interesting read.
>http://www.nytimes.com/2012/11/01/technology/fcc-details-cellphone-problem
>s
>.html
>
>
>And then of course there's this:
>http://www.cnn.com/2012/10/31/tech/mobile/att-tmobile-networks-sandy/
>
>
>
>As for outages...
>We ourselves have been seeing fluctuations in network traffic to NAC for
>very short durations of time (like a few seconds) but have still kept our
>traffic away from the east coast for now, until we hear nobody is on
>generators anymore.
>
>
>And finally, huge kudos to these guys. Sometimes people forget there are
>real humans who operate data centers, and they do amazing things to
>protect our silly little computers in the event of emergency. Amazing
>pictures.
>http://www.huffingtonpost.com/2012/10/31/sandy-data-center_n_2051659.html
>
>
>--
>Darren Schreiber
>CEO / Co-Founder
>2600hz
>
>
>
>
>
>On 10/31/12 7:11 PM, "Matt Yaklin" <[email protected]> wrote:
>
>>
>>Here is some info I was reading earlier today and thought
>>people on this list might enjoy reading it if they have not
>>already.
>>
>>http://www.webhostingtalk.com/showthread.php?t=1205042
>>
>>http://slashdot.org/topic/datacenter/nyc-data-centers-struggle-to-recover
>>-
>>after-sandy/
>>
>>matt
>>
>>On Wed, 31 Oct 2012, Shripal Daphtary wrote:
>>
>>> Major issues out of 75 broad with Windstream paetec. We have had no
>>>did's
>>> inbound from the for 24 hours. They are completely dark in that
>>>location.
>>> Mind you they are not just ny did's. they're nationwide did's
>>>
>>> First of two updates from Windstream all day
>>>
>>>
>>>
>>> NY2 has been OOS since approx 8:30 last night. As a result of the storm
>>> surge Monday night, basement of the building that NY2 & NY3 are in,
>>>flooded.
>>> Late yesterday afternoon, the pump that moves diesel fuel from the
>>>basement
>>> to where our generators are located on the 18th floor, failed. Shortly
>>> thereafter, our generators ran out of fuel and shut down. Batteries to
>>>NY2
>>> lasted 2+ hours while we unsuccessfully brainstormed alternative
>>>solutions
>>> to getting fuel to the 18th floor. Keep in mind, there is no power in
>>>this
>>> building and most if not all of the emergency lighting is no out, so
>>> flashlights are primary lighting. NY2 is also where majority of
>>>connectivity
>>> feeds through, to include some connectivity to Newark, so when the
>>>batteries
>>> ran down, we lost not only NY2, but effectively lost NY3 and some
>>> connectivity to Newark.
>>>
>>> NY2 5E switch 16th floor
>>> NY3 Soft switch 4th floor
>>> Generators. 18th floor
>>> Basement. Fuel tanks
>>>
>>> We are currently working on 2 solutions in parallel:
>>> 1. Procuring a fuel lift pump and 200' of high pressure hose to pump
>>>our own
>>> fuel from ground level to 18th floor generators. Prime and restart
>>> generators, charge batteries for 30 min and begin rebooting equipment.
>>> 2. Contracted with local vendor to do something similar for us
>>>
>>> Since they are both sub-optimal solutions, we are working both
>>> simultaneously to improve of chances of success. Best case is early
>>> afternoon restoration.
>>>
>>> Building management, who is responsible for providing fuel to us
>>>continues
>>> to pump out the flooded basement in order to expose pumps, assess
>>>situation
>>> with intent of permanent repair/replacement of the main pump
>>>
>>>
>>>
>>>
>>> Sent from my mobile device.
>>>
>>> On Oct 31, 2012, at 8:34 PM, "Hiers, David (DS)" <[email protected]>
>>> wrote:
>>>
>>> How are things going out there? We're seeing pretty broad
>>> failures in both termination and origination with Verizon. If
>>> we route calls out Level3, we can at least call out.
>>>
>>>
>>> Thanks,
>>>
>>>
>>> David
>>>
>>>
>>> ________________________________________
>>>
>>> From: [email protected]
>>> [[email protected]] On Behalf Of Jared Geiger
>>> [[email protected]]
>>>
>>> Sent: Tuesday, October 30, 2012 2:35 PM
>>>
>>> To: [email protected]
>>>
>>> Subject: Re: [VoiceOps] Sandy Watch
>>>
>>>
>>> Looks like TATA NYC1 and 2 came back up. Telia went down.
>>> Belgacom is
>>>
>>> back up but taking the Washington -> Paris route instead of the
>>> NYC ->
>>>
>>> London route.
>>>
>>>
>>> Novatel in NYC went completely down. IPs aren't even being
>>> advertised in BGP.
>>>
>>>
>>> On Tue, Oct 30, 2012 at 11:51 AM, Jared Geiger
>>> <[email protected]> wrote:
>>>
>>> We're connected to 2 others. The one in NJ seems up
>>> and we're
>>>
>>> connected to London as well. Looks like the path
>>> across the Atlantic
>>>
>>> is fine.
>>>
>>>
>>> On Tue, Oct 30, 2012 at 9:41 AM, Colin Brown
>>> <[email protected]> wrote:
>>>
>>> If your connected with Tata hopefully
>>> your connected to at least one of the 13
>>> other gsxes they have.
>>>
>>>
>>> On Oct 30, 2012, at 9:34 AM, Jared
>>> Geiger <[email protected]> wrote:
>>>
>>>
>>> 1) Looks like TATA NYC1 and
>>> NYC2 GSXes are down.
>>>
>>> 2) I got an email saying
>>> Telia only had 6 hours of
>>> power left at 6 AM
>>>
>>> Eastern, so they may go dark
>>> as well at noon.
>>>
>>> 3) Our IP routes to Belgacom
>>> are broken and die in NYC on
>>> nlayer,
>>>
>>> Cogent, and Abovenet.
>>>
>>> 4) Looks like Hypercube in
>>> NYC is down.
>>>
>>> 5) Reliance is going down in
>>> NYC
>>>
>>>
>>> It seems all of our other
>>> trunks are up.
>>>
>>>
>>> On Tue, Oct 30, 2012 at 2:37
>>> AM, Hiers, David (DS)
>>> <[email protected]> wrote:
>>>
>>> Nothing too
>>> surprising so
>>> far...
>>>
>>>
>>> There seems to
>>> be pretty strong
>>> correlation
>>> between flooded
>>> areas with no
>>> electricity and
>>> unregistered
>>> devices. I
>>> probably don't
>>> need to drop a
>>> 650 into a fish
>>> tank to see if
>>> the issue is
>>> reproducible,
>>> but it is oddly
>>> tempting.
>>>
>>>
>>>
>>> David
>>>
>>>
>>>
>>>
>>> ________________________________________
>>>
>>> From:
>>> [email protected]
>>> [[email protected]]
>>> On Behalf Of
>>> Hiers, David
>>> (DS)
>>>
>>> Sent: Monday,
>>> October 29, 2012
>>> 11:58 AM
>>>
>>> To:
>>> [email protected]
>>>
>>> Subject:
>>> [VoiceOps] Sandy
>>> Watch
>>>
>>>
>>> Hi,
>>>
>>>
>>> Is anyone seeing
>>> any storm
>>> related issues
>>> yet? I've got
>>> the expected
>>> drop in calls as
>>> the NE shuts
>>> down, not much
>>> more on the
>>> radar.
>>>
>>>
>>> I'll be
>>> watching/updating
>>> this list (and
>>> outages) tonight
>>> for any
>>> significant
>>> events.
>>>
>>>
>>> Thanks,
>>>
>>>
>>> David
>>>
>>>
>>> PS: Having your
>>> machines prepped
>>> for disaster is
>>> good, but what
>>> about your
>>> people? All
>>> that expensive
>>> gear might not
>>> be not worth
>>> much if you've
>>> got no one to
>>> operate it when
>>> the big one
>>> hits. After
>>> all, even first
>>> responders don't
>>> respond if doing
>>> so would imperil
>>> their
>>> families... are
>>> your people
>>> wired in to
>>> ready.gov?
>>>
>>>
>>>
>>>
>>> ________________________________
>>>
>>>
>>> ________________________________
>>>
>>> This message and
>>> any attachments
>>> are intended
>>> only for the use
>>> of the addressee
>>> and may contain
>>> information that
>>> is privileged
>>> and
>>> confidential. If
>>> the reader of
>>> the message is
>>> not the intended
>>> recipient or an
>>> authorized
>>> representative
>>> of the intended
>>> recipient, you
>>> are hereby
>>> notified that
>>> any
>>> dissemination of
>>> this
>>> communication is
>>> strictly
>>> prohibited. If
>>> you have
>>> received this
>>> communication in
>>> error, please
>>> notify us
>>> immediately by
>>> e-mail and
>>> delete the
>>> message and any
>>> attachments from
>>> your system.
>>>
>>>
>>>
>>>
>>> This message and
>>> any attachments
>>> are intended
>>> only for the use
>>> of the addressee
>>> and may contain
>>> information that
>>> is privileged
>>> and
>>> confidential. If
>>> the reader of
>>> the message is
>>> not the intended
>>> recipient or an
>>> authorized
>>> representative
>>> of the intended
>>> recipient, you
>>> are hereby
>>> notified that
>>> any
>>> dissemination of
>>> this
>>> communication is
>>> strictly
>>> prohibited. If
>>> you have
>>> received this
>>> communication in
>>> error, please
>>> notify us
>>> immediately by
>>> e-mail and
>>> delete the
>>> message and any
>>> attachments from
>>> your system.
>>>
>>>
>>>
>>>_______________________________________________
>>>
>>> VoiceOps mailing
>>> list
>>>
>>> [email protected]
>>>
>>>
>>>https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>>
>>> _______________________________________________
>>>
>>> VoiceOps mailing list
>>>
>>> [email protected]
>>>
>>>
>>>https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>>
>>> _______________________________________________
>>>
>>> VoiceOps mailing list
>>>
>>> [email protected]
>>>
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>>
>>> _______________________________________________
>>>
>>> VoiceOps mailing list
>>>
>>> [email protected]
>>>
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>>
>>>
>>> This message and any attachments are intended only for the use
>>> of the addressee and may contain information that is privileged
>>> and confidential. If the reader of the message is not the
>>> intended recipient or an authorized representative of the
>>> intended recipient, you are hereby notified that any
>>> dissemination of this communication is strictly prohibited. If
>>> you have received this communication in error, please notify us
>>> immediately by e-mail and delete the message and any attachments
>>> from your system.
>>>
>>>
>>> _______________________________________________
>>>
>>> VoiceOps mailing list
>>>
>>> [email protected]
>>>
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>>
>
>
>_______________________________________________
>VoiceOps mailing list
>[email protected]
>https://puck.nether.net/mailman/listinfo/voiceops
------------------------------
Message: 2
Date: Wed, 31 Oct 2012 23:19:52 -0500
From: "Frank Bulk" <[email protected]>
To: "'Shripal Daphtary'" <[email protected]>, "Hiers, David \(DS\)"
<[email protected]>
Cc: [email protected]
Subject: Re: [VoiceOps] Sandy Watch
Message-ID: <[email protected]>
Content-Type: text/plain; charset="utf-8"
Anyone know how it is that the diesel tanks in that basement didn?t get
polluted with water? Or is the air intake for the tank above water level?
Frank
From: [email protected] [mailto:[email protected]] On
Behalf Of Shripal Daphtary
Sent: Wednesday, October 31, 2012 7:44 PM
To: Hiers, David (DS)
Cc: [email protected]
Subject: Re: [VoiceOps] Sandy Watch
Major issues out of 75 broad with Windstream paetec. We have had no did's
inbound from the for 24 hours. They are completely dark in that location. Mind
you they are not just ny did's. they're nationwide did's
First of two updates from Windstream all day
NY2 has been OOS since approx 8:30 <x-apple-data-detectors://0> last night. As
a result of the storm surge Monday night <x-apple-data-detectors://1> ,
basement of the building that NY2 & NY3 are in, flooded. Late yesterday
afternoon, the pump that moves diesel fuel from the basement to where our
generators are located on the 18th floor, failed. Shortly thereafter, our
generators ran out of fuel and shut down. Batteries to NY2 lasted 2+ hours
while we unsuccessfully brainstormed alternative solutions to getting fuel to
the 18th floor. Keep in mind, there is no power in this building and most if
not all of the emergency lighting is no out, so flashlights are primary
lighting. NY2 is also where majority of connectivity feeds through, to include
some connectivity to Newark, so when the batteries ran down, we lost not only
NY2, but effectively lost NY3 and some connectivity to Newark.
NY2 5E switch 16th floor
NY3 Soft switch 4th floor
Generators. 18th floor
Basement. Fuel tanks
We are currently working on 2 solutions in parallel:
1. Procuring a fuel lift pump and 200' of high pressure hose to pump our own
fuel from ground level to 18th floor generators. Prime and restart generators,
charge batteries for 30 min and begin rebooting equipment.
2. Contracted with local vendor to do something similar for us
Since they are both sub-optimal solutions, we are working both simultaneously
to improve of chances of success. Best case is early afternoon restoration.
Building management, who is responsible for providing fuel to us continues to
pump out the flooded basement in order to expose pumps, assess situation with
intent of permanent repair/replacement of the main pump
Sent from my mobile device.
On Oct 31, 2012, at 8:34 PM, "Hiers, David (DS)" <[email protected]
<mailto:[email protected]> > wrote:
How are things going out there? We're seeing pretty broad failures in both
termination and origination with Verizon. If we route calls out Level3, we can
at least call out.
Thanks,
David
________________________________________
From: [email protected] <mailto:[email protected]>
[[email protected] <mailto:[email protected]> ] On
Behalf Of Jared Geiger [[email protected] <mailto:[email protected]> ]
Sent: Tuesday, October 30, 2012 2:35 PM
To: [email protected] <mailto:[email protected]>
Subject: Re: [VoiceOps] Sandy Watch
Looks like TATA NYC1 and 2 came back up. Telia went down. Belgacom is
back up but taking the Washington -> Paris route instead of the NYC ->
London route.
Novatel in NYC went completely down. IPs aren't even being advertised in BGP.
On Tue, Oct 30, 2012 at 11:51 AM, Jared Geiger <[email protected]
<mailto:[email protected]> > wrote:
We're connected to 2 others. The one in NJ seems up and we're
connected to London as well. Looks like the path across the Atlantic
is fine.
On Tue, Oct 30, 2012 at 9:41 AM, Colin Brown <[email protected]
<mailto:[email protected]> > wrote:
If your connected with Tata hopefully your connected to at least one of the 13
other gsxes they have.
On Oct 30, 2012, at 9:34 AM, Jared Geiger <[email protected]
<mailto:[email protected]> > wrote:
1) Looks like TATA NYC1 and NYC2 GSXes are down.
2) I got an email saying Telia only had 6 hours of power left at 6 AM
Eastern, so they may go dark as well at noon.
3) Our IP routes to Belgacom are broken and die in NYC on nlayer,
Cogent, and Abovenet.
4) Looks like Hypercube in NYC is down.
5) Reliance is going down in NYC
It seems all of our other trunks are up.
On Tue, Oct 30, 2012 at 2:37 AM, Hiers, David (DS) <[email protected]
<mailto:[email protected]> > wrote:
Nothing too surprising so far...
There seems to be pretty strong correlation between flooded areas with no
electricity and unregistered devices. I probably don't need to drop a 650 into
a fish tank to see if the issue is reproducible, but it is oddly tempting.
David
________________________________________
From: [email protected] <mailto:[email protected]>
[[email protected] <mailto:[email protected]> ] On
Behalf Of Hiers, David (DS)
Sent: Monday, October 29, 2012 11:58 AM
To: [email protected] <mailto:[email protected]>
Subject: [VoiceOps] Sandy Watch
Hi,
Is anyone seeing any storm related issues yet? I've got the expected drop in
calls as the NE shuts down, not much more on the radar.
I'll be watching/updating this list (and outages) tonight for any significant
events.
Thanks,
David
PS: Having your machines prepped for disaster is good, but what about your
people? All that expensive gear might not be not worth much if you've got no
one to operate it when the big one hits. After all, even first responders
don't respond if doing so would imperil their families... are your people wired
in to ready.gov <http://ready.gov> ?
________________________________
________________________________
This message and any attachments are intended only for the use of the addressee
and may contain information that is privileged and confidential. If the reader
of the message is not the intended recipient or an authorized representative of
the intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication
in error, please notify us immediately by e-mail and delete the message and any
attachments from your system.
This message and any attachments are intended only for the use of the addressee
and may contain information that is privileged and confidential. If the reader
of the message is not the intended recipient or an authorized representative of
the intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication
in error, please notify us immediately by e-mail and delete the message and any
attachments from your system.
_______________________________________________
VoiceOps mailing list
[email protected] <mailto:[email protected]>
https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________
VoiceOps mailing list
[email protected] <mailto:[email protected]>
https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________
VoiceOps mailing list
[email protected] <mailto:[email protected]>
https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________
VoiceOps mailing list
[email protected] <mailto:[email protected]>
https://puck.nether.net/mailman/listinfo/voiceops
This message and any attachments are intended only for the use of the addressee
and may contain information that is privileged and confidential. If the reader
of the message is not the intended recipient or an authorized representative of
the intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication
in error, please notify us immediately by e-mail and delete the message and any
attachments from your system.
_______________________________________________
VoiceOps mailing list
[email protected] <mailto:[email protected]>
https://puck.nether.net/mailman/listinfo/voiceops
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://puck.nether.net/pipermail/voiceops/attachments/20121031/753c5abe/attachment.html>
------------------------------
_______________________________________________
VoiceOps mailing list
[email protected]
https://puck.nether.net/mailman/listinfo/voiceops
End of VoiceOps Digest, Vol 41, Issue 1
***************************************
