1) only the namenode is "formatted" and what happens is basically the image file is created and prepped. The image file holds the meta data about how your files are stored on the cluster.
2) The datanodes are not formatted in the conventional sense. Their (datanode) disk usage will grow only when data in the cluster grows when they store data. The moving of file is all automatic and depends on your replication settings (defaulted to 3 nodes). HDFS takes your data, breaks it up into blocks and replicates them across the nodes http://hadoop.apache.org/common/docs/current/hdfs_design.html#Data+Replication . When the Map/Reduce jobs run Hadoop will try to start those jobs on the datanodes where the data already is (so as to not waste time moving the data) otherwise it will move the data to nodes that can do crunching (like if they are idle) to maximize the cluster use. /* Joe Stein http://allthingshadoop.com */ On Sun, May 9, 2010 at 2:49 AM, Vijay Rao <[email protected]> wrote: > Hello, > > I am just reading and understanding Hadoop and all the other components. > However I have a fundamental question for which I am not getting answers in > any of the online material that is out there. > > 1) If hadoop is used then all the slaves and other machines in the cluster > need to be formatted to have HDFS file system. If so what happens to the > tera bytes of data that need to be crunched? Or is the data on a different > machine? > > 2) Everywhere it is mentioned that the main advantage of map/reduce and > hadoop is that it runs on data that is available locally. So does this mean > that once the file system is formatted then I have to move my terabytes of > data and split them across the cluster? > > Thanks > VJ >
