stack wrote:
What did you have configured as the datanode data directory? By default it
writes /tmp. On reboot /tmp is cleared.
Datanodes have two data directories (on separate disks):
$ grep -B1 -A2 data.dir conf/hadoop-site.xml
<property>
<name>dfs.data.dir</name>
<value>/var/lib/hadoop/data1,/var/lib/hadoop/data2</value>
</property>
Files in hadoop seem to be intact. I have tried restarting hadoop and
hbase, but the tables still don't magically reappear as I was hoping
for. What can I do to debug this and hopefully recover my data?
Can you list the content?
./bin/hadoop -fs lsr /hbase
Yes. Data from all seven tables seem to still be there.
$ bin/hadoop fs -dus /hbase
hdfs://bright:9000/hbase 2389548908
$ bin/hadoop fs -lsr /hbase/.META. /hbase/-ROOT-
drwxr-xr-x - hbase supergroup 0 2009-02-27 10:17
/hbase/.META./1028785192
drwxr-xr-x - hbase supergroup 0 2009-02-27 10:17
/hbase/.META./1028785192/historian
drwxr-xr-x - hbase supergroup 0 2009-02-27 10:17
/hbase/.META./1028785192/historian/info
-rw-r--r-- 2 hbase supergroup 9 2009-02-27 10:17
/hbase/.META./1028785192/historian/info/550003861967926036
drwxr-xr-x - hbase supergroup 0 2009-02-27 10:17
/hbase/.META./1028785192/historian/mapfiles
drwxr-xr-x - hbase supergroup 0 2009-02-27 10:17
/hbase/.META./1028785192/historian/mapfiles/550003861967926036
-rw-r--r-- 2 hbase supergroup 3420 2009-02-27 10:17
/hbase/.META./1028785192/historian/mapfiles/550003861967926036/data
-rw-r--r-- 2 hbase supergroup 251 2009-02-27 10:17
/hbase/.META./1028785192/historian/mapfiles/550003861967926036/index
drwxr-xr-x - hbase supergroup 0 2009-02-27 10:17
/hbase/.META./1028785192/info
drwxr-xr-x - hbase supergroup 0 2009-02-27 10:17
/hbase/.META./1028785192/info/info
-rw-r--r-- 2 hbase supergroup 9 2009-02-27 10:17
/hbase/.META./1028785192/info/info/2113823753877423194
drwxr-xr-x - hbase supergroup 0 2009-02-27 10:17
/hbase/.META./1028785192/info/mapfiles
drwxr-xr-x - hbase supergroup 0 2009-02-27 10:17
/hbase/.META./1028785192/info/mapfiles/2113823753877423194
-rw-r--r-- 2 hbase supergroup 3046 2009-02-27 10:17
/hbase/.META./1028785192/info/mapfiles/2113823753877423194/data
-rw-r--r-- 2 hbase supergroup 242 2009-02-27 10:17
/hbase/.META./1028785192/info/mapfiles/2113823753877423194/index
drwxr-xr-x - hbase supergroup 0 2009-02-27 11:16
/hbase/-ROOT-/70236052
drwxr-xr-x - hbase supergroup 0 2009-02-27 11:16
/hbase/-ROOT-/70236052/info
drwxr-xr-x - hbase supergroup 0 2009-02-27 15:12
/hbase/-ROOT-/70236052/info/info
-rw-r--r-- 2 hbase supergroup 9 2009-02-27 15:12
/hbase/-ROOT-/70236052/info/info/293531972055088648
-rw-r--r-- 2 hbase supergroup 11 2009-02-27 10:25
/hbase/-ROOT-/70236052/info/info/5005305484623244108
drwxr-xr-x - hbase supergroup 0 2009-02-27 15:12
/hbase/-ROOT-/70236052/info/mapfiles
drwxr-xr-x - hbase supergroup 0 2009-02-27 15:12
/hbase/-ROOT-/70236052/info/mapfiles/293531972055088648
-rw-r--r-- 2 hbase supergroup 350 2009-02-27 15:12
/hbase/-ROOT-/70236052/info/mapfiles/293531972055088648/data
-rw-r--r-- 2 hbase supergroup 230 2009-02-27 15:12
/hbase/-ROOT-/70236052/info/mapfiles/293531972055088648/index
drwxr-xr-x - hbase supergroup 0 2009-02-27 10:25
/hbase/-ROOT-/70236052/info/mapfiles/5005305484623244108
-rw-r--r-- 2 hbase supergroup 937 2009-02-27 10:25
/hbase/-ROOT-/70236052/info/mapfiles/5005305484623244108/data
-rw-r--r-- 2 hbase supergroup 232 2009-02-27 10:25
/hbase/-ROOT-/70236052/info/mapfiles/5005305484623244108/index
drwxr-xr-x - hbase supergroup 0 2009-02-27 15:12
/hbase/-ROOT-/compaction.dir
Or, did you create the tables in quick succession and before the catalog
tables had a chance to persist -- dump the inmemory edits to the filesystem
-- you restarted? HBase can lose data because there is no working flush in
hdfs currently. Because of this, the commit-log mechanism is ineffective
(Hopefully addressed in hdfs 0.20.0).
I don't think this is the case, data has been written over a couple of
days, so there has been lots of time for persisting it.
