叶双明 wrote:
Thanks for paying attention to my tentative idea!
What I thought isn't how to store the meradata, but the final (or last) way
to recover valuable data in the cluster when something worst (which destroy
the metadata in all multiple NameNode) happen. i.e. terrorist attack or
natural disasters destroy half of cluster nodes within all NameNode, we can
recover as much data as possible by this mechanism, and hava big chance to
recover entire data of cluster because fo original replication.
You want to survive any event that loses a datacentre, you need to
mirror the data off site, chosing that second site with an up to date
fault line map of the city, geological knowledge of where recent
eruptions ended up etc. Which is why nobody builds datacentres in
Enumclaw WA that I'm aware of, the spec for the fabs in/near portland is
they ought to withstand 1-2m of volcanic ash landing on them (what
they'd have got if there'd been an easterly wind when Mount Saint Helens
went). Then once you have some safe location for the second site, talk
to your telco about how the high-bandwidth backbones in your city flow
(Metropolitan Area Ethernet and the like), and try and find somewhere
that meets your requirements.
Then: come up with a protocol that efficiently keeps the two sites up to
date. And reliably: S3 went down last month because they'd been using a
Gossip-style update protocol but wheren't checksumming everything,
because there's no need on a LAN, but of course on a cross-city network
more things can go wrong, and for them it did.
Something to keep multiple hadoop filesystems synchronised efficiently
and reliably across sites could be very useful to many people.
-steve
