It looks that it can not copy the new checkpoint into the NameNode. Can you copy-past what jstack says?
$ sudo -u hdfs jstack <snn-pid> 2013/12/11 Patai Sangbutsarakum <[email protected]> > It just happens without changing anything in the cluster. Secondary > namenode node has been working fine until today i notice that in second > namenode log file stop at. > > 2013-12-11 13:17:41,083 INFO org.apache.hadoop.hdfs.server.common.Storage: > Image file of size 3941631662 saved in 61 seconds. > 2013-12-11 13:19:15,446 INFO org.apache.hadoop.hdfs.server.common.Storage: > Image file of size 3941631662 saved in 94 seconds. > 2013-12-11 13:19:29,760 INFO > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Posted URL > primarynn:50070putimage=1&port=50090&machine=snn2&token=-19:1944437056:1306462484827:1386562944000:1386559939508&newChecksum=2ad3e008becbd2cd5196b2b6ba1498ac > > > #date > Wed Dec 11 14:16:30 PST 2013 > > even after 1 hour passed it's not finish doing the checkpoint. looking at > timestamp of current/fsimage.ckpt at primary namenode; it doesn't show > progress in size and timestamp of the file is days ago. > > already tried to clean the current in snn and restart secondarynamenode > process, but SNN still stop at the same spot even thought the snn process > is still exist. > > > -P >
