Jason Venner wrote:
There is probably a better way than this rather brute force destructive method. If this fails, your hdfs is probably not recoverable so use at your own risk.
Just to clarify, I strongly advise against the following method. It is better to find out and properly fix the actual problem. Changing ip address is supposed to work just fine.
Raghu.
1. return all the nodes to the original ip addresses 2. recursively set your replication count to higher than 1 and wait for the replication updates to finish 3. turn hdfs off on REPCOUNT -1 machines in your cluster by running bin/hadoop-damon.sh stop datanode on each of these machines. 4. put the old ip address in the exclude list on the master 5. change the ip addresses on the hdfs disabled machines to the new ip addresses 6. remove or move away the contents of the dfs directory on the machines (the hadoop.tmp.dir property) 7. add each of the machines back into the cluster by bin/hadoop-damon.sh start datanode 8. wait until the replications re-balance the cluster either by running the setrep -R command or by using the balancer.sh command 9. repeat 3-8 until all of the machines are on the new ip address 10. switch over the namenode pvvpr wrote:Hello, We have a cluster that we initially configured using IP addresses instead of hostnames (i.e., all namenode, datanode references are mentioned as IP addresses rather than hostnames. We did this with an intuition that the cluster might work faster). However, our entire IP address subnet is now changing in our center due to reasons beyond our control. When we changed the machine IP address to the new subnet, the datanodes refuse to start. How can I have a clean start without messing my old data? thanks in advance for help. - Prasad Pingali, LTRC, IIIT, Hyderabad.
