Hi, Thanks for your help.
More details are added below.Please let us know if any additional logs required. We have only one cluster (cluster-1) with 1 NN and with 4 DNs and HBase version is 0.94.10 We are using hbase shell to Create, Export snapshots. In short, what we are trying to do is: -Create snapshots for all HBase tables on cluster-1 and exporte to local file system. Here exported data contains archive+hbase-snapshot -Remove everything from Cluster-1 and importe all exported snapshots from local file system to same cluster-1. Working fine without any issues. -Again,after restoring snapshots, create snapshots for all HBase tables on cluster-1 and exporte to local file system Exported data contains only hbase-snapshot but not archive directory. As we don't have archive directory, snapshot restore is failing with FileNotFoundException. Basically snapshot export on restored snapshot data is not working in our case. Detailed Steps are given below: 1.Created snapshots for all HBase tables on cluster-1 snapshot 'tsdb', 'tsdb_snap_backup' snapshot 'tsdb-meta', 'tsdb-meta_snap_backup' snapshot 'tsdb-tree', 'tsdb-tree_snap_backup' snapshot 'tsdb-uid', 'tsdb-uid_snap_backup' 2.Exported above snapshots to same cluster(cluster-1) under hdfs root (/tsdb_full_backup_2013-12-09_23.32.19) hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot tsdb_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19 -mappers 16; hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot tsdb-meta_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19 -mappers 16; hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot tsdb-tree_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19 -mappers 16; hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot tsdb-uid_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19 -mappers 16; 3. Copied /tsdb_full_backup_2013-12-09_23.32.19 to local file system /opt/airwave/lib/hadoop/bin/hadoop fs -get /tsdb_full_backup_2013-12-09_23.32.19 /tmp/ We could see archive+hbase-snapshot directories copied to local file system /tmp/tsdb_full_backup_2013-12-09_23.32.19 4. Formatted cluster-1 Remove everything from Cluster-1.Now its fresh cluster. 5. copied /tmp/tsdb_full_backup_2013-12-09_23.32.19/ to hdfs path (root dir) /opt/airwave/lib/hadoop/bin/hadoop fs -put /tmp/tsdb_full_backup_2013-12-09_23.32.19 / 6.Import snapshots from hdfs path /tsdb_full_backup_2013-12-09_23.32.19 hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot tsdb_snap_backup -copy-to hdfs:/hbase hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot tsdb-meta_snap_backup -copy-to hdfs:/hbase hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot tsdb-tree_snap_backup -copy-to hdfs:/hbase hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot tsdb-uid_snap_backup -copy-to hdfs:/hbase 7.disable all HBase tables disable 'tsdb' disable'tsdb-meta' disable 'tsdb-tree' disable 'tsdb-uid' 8.restore all HBase tables from snapshot restore_snapshot 'tsdb' restore_snapshot 'tsdb-meta' restore_snapshot 'tsdb-tree' restore_snapshot 'tsdb-uid' 9.enable all HBase tables enable 'tsdb' enable'tsdb-meta' enable 'tsdb-tree' enable 'tsdb-uid' 10.Deleted all snapshots 11.Verified HBase data is imported properly or not. data is imported properly.No issues found. 12.Now again repeat step 1 to step 11 on imported data. In this case hbase-snapshot directory only exported to local file system (eg : /tmp/tsdb_full_backup_2013-12-09_23.42.19) But we dont see archive directory in local file system (eg : /tmp/tsdb_full_backup_2013-12-09_23.42.19) why? Thanks -OC On Mon, Dec 9, 2013 at 5:08 PM, Matteo Bertozzi <[email protected]>wrote: > can you clarify your steps, maybe adding the commands that you used? > Which hbase version are you using? (HBASE-8455 doesn't seems to be related > from the log) > > I assume: > 1) Create, Export and Verify snapshot means > - from hbase shell/client code on Cluster-1 take the snapshot: snapshot > 'table_name', 'snapshotName' > - from a shell of Cluster-1 export snapshot on Cluster-2 with: hbase > org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapshotName ... > - from a shell on Cluster-2 verify snapshot with: hbase > org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot snapshotName > 2) Started HBase from scratch means > - Remove everything from Cluster 1 with: disable 'tableName' .. drop > 'tableName' .. delete_snapshot 'snapshotName' > 3) Import snapshot created in Step 1 means > - from a shell in Cluster-2 export snapshot to Cluster-1: hbase > org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapshotName ... > - from a shell on Cluster-1 verify snapshot with: hbase > org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot snapshotName > - from a shell on Cluster-1: clone_snapshot 'snapshotName', 'tableName' > 4 and 5 I've no idea what you mean > > Matteo > > > > On Mon, Dec 9, 2013 at 11:25 AM, oc tsdb <[email protected]> wrote: > > > Hi, > > > > we are taking tsdb data backup using HBase snapshots. > > > > We tested the scenario as mentioned below: > > > > 1. Created snapshots and exported all snapshots successfully. > > Exported data contains archive+hbase-snapshot > > 2. Started HBase from scratch (removed all Hbase tabled and created > again) > > 3. Successfully imported all snapshots created in step 1. > > 4. Again Created snapshots on imported data (step 3) and exported all > > snapshots. > > Exported data contains only hbase-snapshot but not archive.Why? > > 5. Now tried to import snapshots created in step 4 but we are getting > below > > exception. > > > > Exception in thread "main" java.io.FileNotFoundException: Unable to open > > link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs:// > > > > > tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e > > , > > hdfs:// > > > > > tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.tmp/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e > > , > > hdfs:// > > > > > tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.archive/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e > > ] > > at org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390) > > at > > > > > org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115) > > at > > > > > org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81) > > at > > > > > org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116) > > at > > > > > org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709) > > > > > > Exception in thread "main" java.io.FileNotFoundException: Unable to open > > link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs:// > > > > > tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c > > , > > hdfs:// > > > > > tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c > > , > > hdfs:// > > > > > tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c > > ] > > at org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390) > > at > > > > > org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115) > > at > > > > > org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81) > > at > > > > > org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116) > > at > > > > > org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709) > > > > > > Can someone help us to understand the reason to fail snapshot import. > > Is this issue similar to HBASE-8455? > > > > Thanks > > -OC > > >
