Steve, Would the High Availability<https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html> features for HDFS suit your needs? The method discussed at the link uses JournalNode daemons to keep the Standby Namenode up to date, which would provide your backup.
Paul From: Steven Rand [mailto:[email protected]] Sent: Friday, June 17, 2016 1:45 PM To: [email protected] Subject: best practices for backing up HDFS metadata? Hi all, I'm wondering what the best practices are for backing up HDFS metadata, i.e., the data inside the directories specified by dfs.namenode.name.dir. I'd like to be able to recover from both the loss of all of those directories and from formatting of the NameNode. I'm interested in backing up: * Either the in-memory FSImage, or the most recent on-disk FSImage plus the edit log for all subsequent transactions * The last seen transaction file * The VERSION file * Optionally a checksum of the FSImage that I'm backing up (optionally because I can generate that myself after the fact) It seems like there are basically two options: * Simply make a copy of one of the directories in dfs.namenode.name.dir on the active NameNode while it is running. This way I would get the most recent on-disk FSImage, the edit logs after that FSImage, the last seen transaction file, the VERSION file, and the checksum. * Checkpoint the active NameNode, either using a Secondary NameNode, a Checkpoint Node, a Backup Node, a standby NameNode in an HA configuration, or just by putting the active NameNode in safemode and running hdfs dfsadmin -saveNamespace. Then grab the up to date on-disk FSImage, along with the last seen transaction file, the VERSION file, and optionally the checksum of the FSImage. Is one option any better than the other? The second option seems cleaner, in that I don't have to worry about backing up any edit logs. However, it also requires either putting the NameNode in safemode to checkpoint, or deploying another service to do the checkpointing. The first option feels a bit strange in that I could be taking a backup in the middle of an application writing to HDFS; however, that seems to be true of the second option as well. I'm curious as to how people think about backing up HDFS metadata, and what best practices the community have developed over time. Any thoughts are much appreciated. Thanks, Steve
