Google apparently does the same with GFS / MR, at least by my reading of the Megastore paper.
On Tue, Apr 5, 2011 at 9:59 AM, Alan Gates <[email protected]> wrote: > The concern I have with that approach is I don't think you can guarantee > that Hadoop will never assign tasks to read from the geographically > distributed nodes. At Yahoo we have separate Hadoop clusters in separate > geographic locations and use tools such as distcp to move data between them. > > Alan. > > > On Apr 5, 2011, at 6:14 AM, [email protected] wrote: > > I wish to have a Hadoop cluster with DR (Disaster Recovery), for which I >> need to have the data backup at a geographically different location. (if >> there's an eqrthquake or tsunami that hits one location). >> >> I was thinking along these lines - Make Hadoop rack-aware, set a >> replication factor of 4, and have geographically separated nodes under a >> single Hadoop namespace, and make sure that atleast one replica gets copied >> to the other data centre. Basically, I want to have an Active-Active kind of >> a setup. Is this a feasible idea? If not, why? >> >> Assume that bandwidth is not a limitation - I can have unlimited bandwidth >> - DR is the primary concern - data just cannot be lost. >> >> If this idea is not good, what is the best way to have an Active-Active DR >> Hadoop setup? >> >> This question may not be PIG related, but I thought folks here can suggest >> some good ideas. I am posting the same question on the Hadoop General >> mailing list also. >> >> >> >> Please do not print this email unless it is absolutely necessary. >> >> The information contained in this electronic message and any attachments >> to this message are intended for the exclusive use of the addressee(s) and >> may contain proprietary, confidential or privileged information. If you are >> not the intended recipient, you should not disseminate, distribute or copy >> this e-mail. Please notify the sender immediately and destroy all copies of >> this message and any attachments. >> >> WARNING: Computer viruses can be transmitted via email. The recipient >> should check this email and any attachments for the presence of viruses. The >> company accepts no liability for any damage caused by any virus transmitted >> by this email. >> >> www.wipro.com >> > >
