Hi, thank you for the follow up. The problem is related to a bug in the name resolution of a "clone" (that in your case the is the restore of a snapshot of a restored table). This problem was fixed as part of HBASE-8760, which should be integrated in 0.94.12
Matteo On Tue, Dec 10, 2013 at 10:00 AM, oc tsdb <[email protected]> wrote: > Hi, > > Thanks for your help. > > More details are added below.Please let us know if any additional logs > required. > > We have only one cluster (cluster-1) with 1 NN and with 4 DNs and HBase > version is 0.94.10 > > We are using hbase shell to Create, Export snapshots. > > In short, what we are trying to do is: > > -Create snapshots for all HBase tables on cluster-1 and exporte to local > file system. > Here exported data contains archive+hbase-snapshot > -Remove everything from Cluster-1 and importe all exported snapshots from > local file system to same cluster-1. > Working fine without any issues. > -Again,after restoring snapshots, create snapshots for all HBase tables on > cluster-1 and exporte to local file system > Exported data contains only hbase-snapshot but not archive directory. > As we don't have archive directory, snapshot restore is failing with > FileNotFoundException. > > Basically snapshot export on restored snapshot data is not working in our > case. > > Detailed Steps are given below: > > 1.Created snapshots for all HBase tables on cluster-1 > > snapshot 'tsdb', 'tsdb_snap_backup' > snapshot 'tsdb-meta', 'tsdb-meta_snap_backup' > snapshot 'tsdb-tree', 'tsdb-tree_snap_backup' > snapshot 'tsdb-uid', 'tsdb-uid_snap_backup' > > 2.Exported above snapshots to same cluster(cluster-1) under hdfs root > (/tsdb_full_backup_2013-12-09_23.32.19) > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > tsdb_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19 -mappers > 16; > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > tsdb-meta_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19 > -mappers 16; > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > tsdb-tree_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19 > -mappers 16; > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > tsdb-uid_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19 > -mappers 16; > > > 3. Copied /tsdb_full_backup_2013-12-09_23.32.19 to local file system > > /opt/airwave/lib/hadoop/bin/hadoop fs -get > /tsdb_full_backup_2013-12-09_23.32.19 /tmp/ > > We could see archive+hbase-snapshot directories copied to local file system > /tmp/tsdb_full_backup_2013-12-09_23.32.19 > > 4. Formatted cluster-1 > Remove everything from Cluster-1.Now its fresh cluster. > > 5. copied /tmp/tsdb_full_backup_2013-12-09_23.32.19/ to hdfs path (root > dir) > > /opt/airwave/lib/hadoop/bin/hadoop fs -put > /tmp/tsdb_full_backup_2013-12-09_23.32.19 / > > 6.Import snapshots from hdfs path /tsdb_full_backup_2013-12-09_23.32.19 > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D > hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot > tsdb_snap_backup -copy-to hdfs:/hbase > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D > hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot > tsdb-meta_snap_backup -copy-to hdfs:/hbase > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D > hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot > tsdb-tree_snap_backup -copy-to hdfs:/hbase > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D > hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot > tsdb-uid_snap_backup -copy-to hdfs:/hbase > > 7.disable all HBase tables > disable 'tsdb' > disable'tsdb-meta' > disable 'tsdb-tree' > disable 'tsdb-uid' > > 8.restore all HBase tables from snapshot > > > restore_snapshot 'tsdb' > restore_snapshot 'tsdb-meta' > restore_snapshot 'tsdb-tree' > restore_snapshot 'tsdb-uid' > > 9.enable all HBase tables > enable 'tsdb' > enable'tsdb-meta' > enable 'tsdb-tree' > enable 'tsdb-uid' > > 10.Deleted all snapshots > > 11.Verified HBase data is imported properly or not. > data is imported properly.No issues found. > > 12.Now again repeat step 1 to step 11 on imported data. > > In this case hbase-snapshot directory only exported to local file system > (eg : /tmp/tsdb_full_backup_2013-12-09_23.42.19) > But we dont see archive directory in local file system (eg : > /tmp/tsdb_full_backup_2013-12-09_23.42.19) why? > > Thanks > -OC > > > On Mon, Dec 9, 2013 at 5:08 PM, Matteo Bertozzi <[email protected] > >wrote: > > > can you clarify your steps, maybe adding the commands that you used? > > Which hbase version are you using? (HBASE-8455 doesn't seems to be > related > > from the log) > > > > I assume: > > 1) Create, Export and Verify snapshot means > > - from hbase shell/client code on Cluster-1 take the snapshot: snapshot > > 'table_name', 'snapshotName' > > - from a shell of Cluster-1 export snapshot on Cluster-2 with: hbase > > org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapshotName > ... > > - from a shell on Cluster-2 verify snapshot with: hbase > > org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot snapshotName > > 2) Started HBase from scratch means > > - Remove everything from Cluster 1 with: disable 'tableName' .. drop > > 'tableName' .. delete_snapshot 'snapshotName' > > 3) Import snapshot created in Step 1 means > > - from a shell in Cluster-2 export snapshot to Cluster-1: hbase > > org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapshotName > ... > > - from a shell on Cluster-1 verify snapshot with: hbase > > org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot snapshotName > > - from a shell on Cluster-1: clone_snapshot 'snapshotName', 'tableName' > > 4 and 5 I've no idea what you mean > > > > Matteo > > > > > > > > On Mon, Dec 9, 2013 at 11:25 AM, oc tsdb <[email protected]> wrote: > > > > > Hi, > > > > > > we are taking tsdb data backup using HBase snapshots. > > > > > > We tested the scenario as mentioned below: > > > > > > 1. Created snapshots and exported all snapshots successfully. > > > Exported data contains archive+hbase-snapshot > > > 2. Started HBase from scratch (removed all Hbase tabled and created > > again) > > > 3. Successfully imported all snapshots created in step 1. > > > 4. Again Created snapshots on imported data (step 3) and exported all > > > snapshots. > > > Exported data contains only hbase-snapshot but not archive.Why? > > > 5. Now tried to import snapshots created in step 4 but we are getting > > below > > > exception. > > > > > > Exception in thread "main" java.io.FileNotFoundException: Unable to > open > > > link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs:// > > > > > > > > > tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e > > > , > > > hdfs:// > > > > > > > > > tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.tmp/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e > > > , > > > hdfs:// > > > > > > > > > tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.archive/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e > > > ] > > > at org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376) > > > at > > > > > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390) > > > at > > > > > > > > > org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115) > > > at > > > > > > > > > org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81) > > > at > > > > > > > > > org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116) > > > at > > > > > > > > > org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101) > > > at > > > > > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385) > > > at > > > > > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633) > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > > > at > > > > > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705) > > > at > > > > > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709) > > > > > > > > > Exception in thread "main" java.io.FileNotFoundException: Unable to > open > > > link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs:// > > > > > > > > > tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c > > > , > > > hdfs:// > > > > > > > > > tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c > > > , > > > hdfs:// > > > > > > > > > tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c > > > ] > > > at org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376) > > > at > > > > > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390) > > > at > > > > > > > > > org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115) > > > at > > > > > > > > > org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81) > > > at > > > > > > > > > org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116) > > > at > > > > > > > > > org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101) > > > at > > > > > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385) > > > at > > > > > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633) > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > > > at > > > > > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705) > > > at > > > > > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709) > > > > > > > > > Can someone help us to understand the reason to fail snapshot import. > > > Is this issue similar to HBASE-8455? > > > > > > Thanks > > > -OC > > > > > >
