Re: Fire, Power loss at Fisher Plaza in Seattle
Multiple folks on Twitter who are in the area are reporting a 5-6 hour ETA. -Joe -- Joe Richards j...@disconformity.net -- ipv4: http://www.disconformity.net [ 72.29.169.48/28 ] ipv6: http://ipv6.disconformity.net [ 2001:48c0:1001:1::/64 ] blog: http://www.mainlined.org
Re: Fire, Power loss at Fisher Plaza in Seattle
Darren Bolding wrote: Interestingly, this building is also the production studios for several Seattle TV and radio stations. There is no ETA for resolution. Apparently it took authorize.net with it, too: http://twitter.com/authorizenet ~Seth
RE: Fire, Power loss at Fisher Plaza in Seattle
This begs the question of what basic parameters should be for a carrier hotel or co-lo. Given that we're getting designated Critical Infrastructure, we'd getter start coming up with some, or we'll have them defined for us. The old NEBS standards were too much of a straightjacket, but the current situation, where any buffoon who wants to can claim to be something they aren't (redundant and reliable) undermines the business of those who actually spend the money, and make the effort, to provide a true carrier grade co-lo. This is life in the current Internet: Overpromise, and Underdeliver. -Original Message- From: David Hubbard [mailto:dhubb...@dino.hostasaurus.com] Sent: Friday, July 03, 2009 10:05 AM To: NANOG list Subject: RE: Fire, Power loss at Fisher Plaza in Seattle From: Seth Mattinen [mailto:se...@rollernet.us] Apparently it took authorize.net with it, too: http://twitter.com/authorizenet ~Seth No technical explanation of course but it also took down their 'backup facility' according to them on twitter; I assume some bad routing/DNS if they do actually have a backup facility. Lots of online stores are offline right now because of this, and the holiday is unfortunately keeping those store owners from knowing they are not making sales right now. Life in ecommerce... David
Re: Fire, Power loss at Fisher Plaza in Seattle
Tomas L. Byrnes wrote: This begs the question of what basic parameters should be for a carrier hotel or co-lo. Given that we're getting designated Critical Infrastructure, we'd getter start coming up with some, or we'll have them defined for us. The old NEBS standards were too much of a straightjacket, but the current situation, where any buffoon who wants to can claim to be something they aren't (redundant and reliable) undermines the business of those who actually spend the money, and make the effort, to provide a true carrier grade co-lo. Absolutely. Then your pricing is so far out of whack with the apparent competition that it's hard to get customers when it appears one can get the same/better for far less. Me, personally, I just don't say things like 100% uptime or claim to be a carrier-grade facility. But I think that scares people off when my competitors (and I've seen the insides of some of the horrid trash heaps they call a NOC) claim they do. This is life in the current Internet: Overpromise, and Underdeliver. Our flywheel systems are so failure-proof and thinking outside the box that we don't need a silly battery UPS that can cold-start! I know outages and related discussion end up attracting the off-topic hammer here on NANOG, but I do find them interesting and worthwhile. ~Seth
RE: Fire, Power loss at Fisher Plaza in Seattle
http://seattletimes.nwsource.com/html/localnews/2009415571_apwafisherpla zafire1stldwritethru.html -Original Message- From: David Hubbard [mailto:dhubb...@dino.hostasaurus.com] Sent: Friday, July 03, 2009 1:05 PM To: NANOG list Subject: RE: Fire, Power loss at Fisher Plaza in Seattle From: Seth Mattinen [mailto:se...@rollernet.us] Apparently it took authorize.net with it, too: http://twitter.com/authorizenet ~Seth No technical explanation of course but it also took down their 'backup facility' according to them on twitter; I assume some bad routing/DNS if they do actually have a backup facility. Lots of online stores are offline right now because of this, and the holiday is unfortunately keeping those store owners from knowing they are not making sales right now. Life in ecommerce... David
Re: Fire, Power loss at Fisher Plaza in Seattle
On Fri, Jul 3, 2009 at 1:20 PM, Tomas L. Byrnest...@byrneit.net wrote: This begs the question of what basic parameters should be for a carrier hotel or co-lo. [...] The old NEBS standards were too much of a straightjacket. Tomas, There is a useful standard: ANSI/TIA-942. It offers specifications for four tiers of data centers ranging from tier 1 (a basic data center with no redundancy) to tier 4 (fully fault tolerant). http://www.tiaonline.org/standards/catalog/search.cfm?standards_criteria=TIA-942 (the 2005 one) Judging from http://www.techlinks.net/community/articles/article/1-article-submission-forms/14833-a-quick-primer-on-data-center-tier-classifications there's even research that projects what sort of annual downtime you can expect for each of the tiers described by the standard. When I walk into a data center, I make a habit of asking which tier they achieve, at least for the HVAC and electrical systems. And then I ask to see the components which the tier claim says they should have. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
RE: Fire, Power loss at Fisher Plaza in Seattle
-Original Message- From: Tomas L. Byrnes [mailto:t...@byrneit.net] Sent: Fri 7/3/2009 10:20 AM To: David Hubbard; NANOG list Subject: RE: Fire, Power loss at Fisher Plaza in Seattle This begs the question of what basic parameters should be for a carrier hotel or co-lo. Given that we're getting designated Critical Infrastructure, we'd getter start coming up with some, or we'll have them defined for us. I think the more important question is, what do you consider redundancy? We have facilities in Plaza East (no down) and Plaza West (unaffected). If you are critical infrastructure there is no amount of redundancy that you should offload onto a colo provider. Instead, you build your redundancy across different data centers, different providers, different everything. If you rely on a single provider for any of the aforementioned then you have built in at least one single point of failure, regardless of the resiliency of the underlying provider. My .02, worth almost every penny. Mike
Re: Fire, Power loss at Fisher Plaza in Seattle
On Fri, 3 Jul 2009, William Herrin wrote: There is a useful standard: ANSI/TIA-942. It offers specifications for four tiers of data centers ranging from tier 1 (a basic data center with no redundancy) to tier 4 (fully fault tolerant). Are you better off with a single tier 4 data center, multiple tier 1 data centers, or something in between? Distance and quantity versus complexity and scaling versus cost and risk. Sometimes no matter what you choose, you might be wrong. Earth is a single point of failure, where is your backup site?
Re: Fire, Power loss at Fisher Plaza in Seattle
Earth is a single point of failure, where is your backup site? This reminds me of the 1996 thread about how MAE-East still had no generator. Same topic, roughly, some of the same people (hi, Sean). Sure, the line about the Earth SPOF is catchy, but in terms of more likely scenarios: how many people stand *outside* the tier 4 datacenter and imagine a fire marshal pointing at the building and saying, Turn *that* off, now. I've seen that happen a couple times since the WilTel POP thing in 1996. Stephen
Re: Fire, Power loss at Fisher Plaza in Seattle
Wasn't Authorize.net affected by this? We received a support ticket about why Authorize.net is down today (I don't know either, I don't ask too many questions). Jeff On Fri, Jul 3, 2009 at 4:21 PM, Tomas L. Byrnest...@byrneit.net wrote: Earth is a single point of failure, where is your backup site? [TLB:] Given that all my customers are on Earth, I don't need one if my customers also are down. -- Jeffrey Lyon, Leadership Team jeffrey.l...@blacklotus.net | http://www.blacklotus.net Black Lotus Communications of The IRC Company, Inc. Look for us at HostingCon 2009 in Washington, DC on August 10th - 12th at Booth #401.
Re: Fire, Power loss at Fisher Plaza in Seattle
Yes it was. On Jul 3, 2009, at 4:29 PM, Jeffrey Lyon wrote: Wasn't Authorize.net affected by this? We received a support ticket about why Authorize.net is down today (I don't know either, I don't ask too many questions). Jeff On Fri, Jul 3, 2009 at 4:21 PM, Tomas L. Byrnest...@byrneit.net wrote: Earth is a single point of failure, where is your backup site? [TLB:] Given that all my customers are on Earth, I don't need one if my customers also are down. -- Jeffrey Lyon, Leadership Team jeffrey.l...@blacklotus.net | http://www.blacklotus.net Black Lotus Communications of The IRC Company, Inc. Look for us at HostingCon 2009 in Washington, DC on August 10th - 12th at Booth #401.
Re: Fire, Power loss at Fisher Plaza in Seattle
Power to some of the affected sections of the building has been restored via existing onsite generators. The central power risers cannot be connected to current generators in a timely manner due to excessive damage to the electrical switching equipment (and those generators may still be in standing water). These provide power to a number of colocated systems. Temporary generators are on order to be connected to the central risers, and the site expects that to be complete sometime late this evening. As best I can tell, there is still no utility power connected to any of the systems. The AC systems (chiller and crac) are currently not working. It is not clear to me whether these will be brought back on line when the temporary generators are available, but I am assuming so. It was pleasant to see the general positive attitude, sharing of information and offers of assistance that were made by representatives of the various tenants, customers and carriers that were on the scene. The usual suspects (companies and individuals) stepped up and took care of things, as they always seem to. --D On Fri, Jul 3, 2009 at 1:39 PM, Leo Bicknell bickn...@ufp.org wrote: In a message written on Fri, Jul 03, 2009 at 03:22:14PM -0400, Sean Donelan wrote: Are you better off with a single tier 4 data center, multiple tier 1 data centers, or something in between? It depends entirely on your dependency on connectivity. One extreme is something like a Central Office. Lots of cables from end-sites terminate in the building. Having a duplicate of the head end termination equipment on the opposite coast is darn near useless. If the building goes down, the users going through it go down. Tier 4 is probably a good idea. The other extreme is a pure content play (YouTube, Google Search). Users don't care which data center they hit (within reason), and indeed often don't know. You're better off having data centers spread out all over, both so you're more likely to only loose one at a time, but also so that the loss of one is relatively unimportant. Once you're already in this architecture, Tier 1 is generally cheaper. There are two problems though. First, most folks don't fit neatly in one of these buckets. They have some ties to local infrastructure, and some items which are not tied. Latency as a performance penality is very subjective. A backup 1000 miles away is fine for many things, and very bad for some things. Second, most folks don't have choices. It would be nice if most cities had three each Tier 1, 2, 3 and 4 data centers available so there was choice and competition but that's rare. Very few companies consider these choices rationally; often because choices are made by different groups. I am amazed how many times inside of an ISP the folks deploying the DNS and mail servers are firewalled from the folks deploying the network, to the point where you have to get to the President to reach common management. This leads to them making choices in opposite directions that end up costing extra money the company, and often resulting in a much lower uptimes than expected. Having the network group deploy a single point of failure to the Tier 4 data center the server guys required is, well, silly. However, more important than all of this is testing your infrastructure. Would you feel comfortable walking into your data center and ripping the power cable out of some bit of equipment at random _right now_? If not, you have no faith your equipment will work in an outage. -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ -- -- Darren Bolding -- -- dar...@bolding.org --
Re: Fire, Power loss at Fisher Plaza in Seattle
On Jul 3, 2009, at 4:29 PM, Ben Carleton wrote: Yes it was. On Jul 3, 2009, at 4:29 PM, Jeffrey Lyon wrote: Wasn't Authorize.net affected by this? We received a support ticket about why Authorize.net is down today (I don't know either, I don't ask too many questions). Authorize.net was for a while completely off the air, and companies that relied upon them were not getting credit card authorizations (and, thus, no ecommerce). I think it is still only partially functional. Authorize.net has been communicating with customers mostly (entirely ?) with twitter - they are @AuthorizeNet with a hash tab of #authorizenet If you go there, you will see a lot of status messages like #authorizenet (cont.) Do not manually submit ARB transactions b/c you run the risk of your merchants being double billed. 10 minutes ago from web (i.e., 4:47 EDT). You will also see a lot of posts from annoyed people if you search on #authorizenet It's an interesting use of Web 2.0 for emergency communications. Regards Marshall Jeff On Fri, Jul 3, 2009 at 4:21 PM, Tomas L. Byrnest...@byrneit.net wrote: Earth is a single point of failure, where is your backup site? [TLB:] Given that all my customers are on Earth, I don't need one if my customers also are down. -- Jeffrey Lyon, Leadership Team jeffrey.l...@blacklotus.net | http://www.blacklotus.net Black Lotus Communications of The IRC Company, Inc. Look for us at HostingCon 2009 in Washington, DC on August 10th - 12th at Booth #401.
Re: Fire, Power loss at Fisher Plaza in Seattle
That's a great idea, use some lame Web 2.0 trend to communicate with actual real life customers. /sarcasm Jeff On Fri, Jul 3, 2009 at 4:59 PM, Marshall Eubankst...@americafree.tv wrote: On Jul 3, 2009, at 4:29 PM, Ben Carleton wrote: Yes it was. On Jul 3, 2009, at 4:29 PM, Jeffrey Lyon wrote: Wasn't Authorize.net affected by this? We received a support ticket about why Authorize.net is down today (I don't know either, I don't ask too many questions). Authorize.net was for a while completely off the air, and companies that relied upon them were not getting credit card authorizations (and, thus, no ecommerce). I think it is still only partially functional. Authorize.net has been communicating with customers mostly (entirely ?) with twitter - they are @AuthorizeNet with a hash tab of #authorizenet If you go there, you will see a lot of status messages like #authorizenet (cont.) Do not manually submit ARB transactions b/c you run the risk of your merchants being double billed. 10 minutes ago from web (i.e., 4:47 EDT). You will also see a lot of posts from annoyed people if you search on #authorizenet It's an interesting use of Web 2.0 for emergency communications. Regards Marshall Jeff On Fri, Jul 3, 2009 at 4:21 PM, Tomas L. Byrnest...@byrneit.net wrote: Earth is a single point of failure, where is your backup site? [TLB:] Given that all my customers are on Earth, I don't need one if my customers also are down. -- Jeffrey Lyon, Leadership Team jeffrey.l...@blacklotus.net | http://www.blacklotus.net Black Lotus Communications of The IRC Company, Inc. Look for us at HostingCon 2009 in Washington, DC on August 10th - 12th at Booth #401. -- Jeffrey Lyon, Leadership Team jeffrey.l...@blacklotus.net | http://www.blacklotus.net Black Lotus Communications of The IRC Company, Inc. Look for us at HostingCon 2009 in Washington, DC on August 10th - 12th at Booth #401.
Re: Fire, Power loss at Fisher Plaza in Seattle
In a message written on Fri, Jul 03, 2009 at 03:22:14PM -0400, Sean Donelan wrote: Are you better off with a single tier 4 data center, multiple tier 1 data centers, or something in between? It depends entirely on your dependency on connectivity. One extreme is something like a Central Office. Lots of cables from end-sites terminate in the building. Having a duplicate of the head end termination equipment on the opposite coast is darn near useless. If the building goes down, the users going through it go down. Tier 4 is probably a good idea. The other extreme is a pure content play (YouTube, Google Search). Users don't care which data center they hit (within reason), and indeed often don't know. You're better off having data centers spread out all over, both so you're more likely to only loose one at a time, but also so that the loss of one is relatively unimportant. Once you're already in this architecture, Tier 1 is generally cheaper. There are two problems though. First, most folks don't fit neatly in one of these buckets. They have some ties to local infrastructure, and some items which are not tied. Latency as a performance penality is very subjective. A backup 1000 miles away is fine for many things, and very bad for some things. Second, most folks don't have choices. It would be nice if most cities had three each Tier 1, 2, 3 and 4 data centers available so there was choice and competition but that's rare. Very few companies consider these choices rationally; often because choices are made by different groups. I am amazed how many times inside of an ISP the folks deploying the DNS and mail servers are firewalled from the folks deploying the network, to the point where you have to get to the President to reach common management. This leads to them making choices in opposite directions that end up costing extra money the company, and often resulting in a much lower uptimes than expected. Having the network group deploy a single point of failure to the Tier 4 data center the server guys required is, well, silly. However, more important than all of this is testing your infrastructure. Would you feel comfortable walking into your data center and ripping the power cable out of some bit of equipment at random _right now_? If not, you have no faith your equipment will work in an outage. -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ pgpWheAQ4NuE8.pgp Description: PGP signature
RE: Fire, Power loss at Fisher Plaza in Seattle
On Fri, 2009-07-03 at 13:21 -0700, Tomas L. Byrnes wrote: Earth is a single point of failure, where is your backup site? [TLB:] Given that all my customers are on Earth, I don't need one if my customers also are down. Bad Day !
Re: Fire, Power loss at Fisher Plaza in Seattle
On Jul 3, 2009, at 5:11 PM, Jeffrey Lyon wrote: That's a great idea, use some lame Web 2.0 trend to communicate with actual real life customers. /sarcasm I would assume they figured it was better than just remaining silent. Regards Marshall Jeff On Fri, Jul 3, 2009 at 4:59 PM, Marshall Eubankst...@americafree.tv wrote: On Jul 3, 2009, at 4:29 PM, Ben Carleton wrote: Yes it was. On Jul 3, 2009, at 4:29 PM, Jeffrey Lyon wrote: Wasn't Authorize.net affected by this? We received a support ticket about why Authorize.net is down today (I don't know either, I don't ask too many questions). Authorize.net was for a while completely off the air, and companies that relied upon them were not getting credit card authorizations (and, thus, no ecommerce). I think it is still only partially functional. Authorize.net has been communicating with customers mostly (entirely ?) with twitter - they are @AuthorizeNet with a hash tab of #authorizenet If you go there, you will see a lot of status messages like #authorizenet (cont.) Do not manually submit ARB transactions b/c you run the risk of your merchants being double billed. 10 minutes ago from web (i.e., 4:47 EDT). You will also see a lot of posts from annoyed people if you search on #authorizenet It's an interesting use of Web 2.0 for emergency communications. Regards Marshall Jeff On Fri, Jul 3, 2009 at 4:21 PM, Tomas L. Byrnest...@byrneit.net wrote: Earth is a single point of failure, where is your backup site? [TLB:] Given that all my customers are on Earth, I don't need one if my customers also are down. -- Jeffrey Lyon, Leadership Team jeffrey.l...@blacklotus.net | http://www.blacklotus.net Black Lotus Communications of The IRC Company, Inc. Look for us at HostingCon 2009 in Washington, DC on August 10th - 12th at Booth #401. -- Jeffrey Lyon, Leadership Team jeffrey.l...@blacklotus.net | http://www.blacklotus.net Black Lotus Communications of The IRC Company, Inc. Look for us at HostingCon 2009 in Washington, DC on August 10th - 12th at Booth #401. Regards Marshall Eubanks CEO / AmericaFree.TV
