It just happens without changing anything in the cluster. Secondary namenode node has been working fine until today i notice that in second namenode log file stop at.
2013-12-11 13:17:41,083 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 3941631662 saved in 61 seconds. 2013-12-11 13:19:15,446 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 3941631662 saved in 94 seconds. 2013-12-11 13:19:29,760 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Posted URL primarynn:50070putimage=1&port=50090&machine=snn2&token=-19:1944437056:1306462484827:1386562944000:1386559939508&newChecksum=2ad3e008becbd2cd5196b2b6ba1498ac #date Wed Dec 11 14:16:30 PST 2013 even after 1 hour passed it's not finish doing the checkpoint. looking at timestamp of current/fsimage.ckpt at primary namenode; it doesn't show progress in size and timestamp of the file is days ago. already tried to clean the current in snn and restart secondarynamenode process, but SNN still stop at the same spot even thought the snn process is still exist. -P
