Aled Jones wrote:
Anyhoo, I'm fairly new to hadoop and was wondering about the redundancy aspects of it. If I have a few servers running for nutch, one being a name and data node, the others just data nodes, what happens when the name node falls over? To get proper redundancy in a hadoop system do I need at least 2 name nodes?
You just need a single namenode. With a large cluster, some machine will fail frequently, but any particular machine is unlikely to fail very frequently. Thus it would be a bad idea to have every node to be critical to system operation, but making a single node critical is acceptable.
Currently the (undocumented) best-practice is to have a crontab entry that uses rsync or similar to regularly copy the namenode's data to another machine. Long-term we hope to have namenode backup built into Hadoop. For example, we might have a hot-spare namenode. All fs changes can be trickled to the hot spare as they are made. Switchover on namenode failure would still be manual (for the reasons described in Googles GFS paper).
Doug
