All hadoop contributors/experts, I am trying to simulate split brain in our installation. There are a few things we want to know
1. Does data corruption happen? 2. If Yes in #1, how to recover from it. 3. What are the corrective steps to take in this situation e.g. killing one namenode etc So to simulate this I took following steps. 1. We already have a healthy test cluster, consisting of 4 machines. One machine runs namenode and a datanode, other machine runs secondarynamenode and a datanode, 3rd runs jobtracker and a datanode, and 4th one just a datanode. 2. Copied the hadoop installation folder to a new location in the datanode. 3. Kept all configurations same in hdfs-site and core-site xmls, except renamed the fs.default.name to a different URI 4. The namenode directory - dfs.name.dir was pointing to the same shared NFS mounted directory to which the main namenode points to. I started this standby namenode using following command bin/hadoop-daemon.sh --config conf --hosts slaves start namenode It errored out saying that "the directory is already locked", which is an expected behaviour. The directory has been locked by the original namenode. So I changed the dfs.name.dir to some other folder, and issued the same command. It fails with message - "namenode has not been formatted", which is also expected. This makes me think - does splitbrain situation really occur in hadoop? My understanding is that split brain happens because of timeouts on the main namenode. The way it happens is, when the timeout occurs, the HA implementation - Be it Linux HA, Veritas etc., thinks that the main namenode has died and tries to start the standby namenode. The standby namenode starts up and then main namenode comes back from the timeout phase and starts functioning as if nothing happened, giving rise to 2 namenodes in the cluster - Split Brain. Considering the error messages and the above understanding, I cannot point 2 different namenodes to same directory, because the main namenode isn't responding but has locked the directory. So can I safely conclude that split brain does not occur in hadoop? Or am I missing any other situation where split brain happens and the namenode directory is not locked, thus allowing the standby namenode also to start up? Has anybody encountered this? Any help is really appreciated. Harshad
