I attempted to restore a backed-up "meta" directory from a copy made by 'hbase hbck', so my directory structure may be messed up:
[user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs -ls -R /hbase/data/hbase drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48 /hbase/data/hbase/meta drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48 /hbase/data/hbase/meta/.tabledesc -rw-r--r-- 2 hbase hadoop 372 2016-01-04 14:48 /hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001 drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48 /hbase/data/hbase/meta/.tmp drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28 /hbase/data/hbase/meta/1588230740 -rw-r--r-- 3 hdfs hadoop 32 2016-01-04 16:28 /hbase/data/hbase/meta/1588230740/.regioninfo drwxr-xr-x - hbase hadoop 0 2016-01-04 14:47 /hbase/data/hbase/meta/1588230740/info drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28 /hbase/data/hbase/meta/1588230740/recovered.edits -rw-r--r-- 3 hdfs hadoop 0 2016-01-04 16:28 /hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid And I'm not seeing any "namespace" directory. Will check out the server now... Shane O. ======================== Shane O'Donnell <http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=6a0326e0-41e4-46fd-bcce-114f44302f69> 4819 Emperor Blvd., Ste 400 Durham, North Carolina 27703 tel: +1.424.262.KNOW x703 skype: shaneodonnell email: [email protected] ======================== :wq! On Mon, Jan 4, 2016 at 9:44 AM, Ted Yu <[email protected]> wrote: > Can you log onto the server hosting region 82432aca9ede964943b40753cb64e808 > and see what happened ? > > See if the namespace table can be found under rootdir. > e.g. assuming /apps/hbase/data is the rootdir, you should see something > similar to the following: > > hdfs dfs -ls /apps/hbase/data/data/hbase/namespace > Found 3 items > drwxr-xr-x - hbase hdfs 0 2015-12-15 19:15 > /apps/hbase/data/data/hbase/namespace/.tabledesc > drwxr-xr-x - hbase hdfs 0 2015-12-15 19:15 > /apps/hbase/data/data/hbase/namespace/.tmp > drwxr-xr-x - hbase hdfs 0 2015-12-15 19:51 > /apps/hbase/data/data/hbase/namespace/844e1bab028e0ecc07d3bd8e34cc76a8 > > On Mon, Jan 4, 2016 at 6:37 AM, Shane O'Donnell <[email protected]> > wrote: > > > Some progress... > > > > /hbase did NOT have either the hbase.id or hbase.version files so I > > temporarily changed hbase.rootdir and started the master so the files > would > > be recreated elsewhere and copied them in. > > > > Now it starts fine, but my hbase tables are gone. Specifically, I'm > > getting this error: > > > > hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808. > > state=FAILED_OPEN, ts=Mon Jan 04 14:28:32 UTC 2016 (18s ago), > > server=ip-10-0-1-29.ec2.internal,60020,1451524442749 > > > > Is my existing data toast, or is there a crafty way out of this? > > > > Thanks - > > > > Shane O. > > > > ======================== > > Shane O'Donnell > > < > > > http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9387b86f-bfad-4ebe-c506-e91a94d0c960 > > > > > 4819 Emperor Blvd., Ste 400 > > Durham, North Carolina 27703 > > tel: +1.424.262.KNOW x703 > > skype: shaneodonnell > > email: [email protected] > > ======================== > > :wq! > > > > On Sun, Jan 3, 2016 at 10:21 PM, Shane O'Donnell <[email protected]> > > wrote: > > > > > Hi - > > > > > > My cluster has been running perfectly until the other day when I found > it > > > down. > > > > > > The error seems to be related not being able to get the ClusterID from > > > zookeeper, but I'm stumped as to what to do about it. This seems to be > > the > > > relevant part of the master's log: > > > > > > http://pastebin.com/C3iaxM3p > > > > > > starting at line 235 (also highlighted). > > > > > > Any help is appreciated! > > > > > > Thanks - > > > > > > Shane O. > > > ======================== > > > Shane O'Donnell > > > > > > < > > > http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac > > > > > > 4819 Emperor Blvd., Ste 400 > > > Durham, North Carolina 27703 > > > tel: +1.424.262.KNOW x703 > > > skype: shaneodonnell > > > email: [email protected] > > > ======================== > > > :wq! > > > > > >
