I believe there was at least one good block (see fsck output). All
data nodes were up at the time according to the web page. I grep'd
the namenode log files for the under replicated blocks and only got
an entry for when it was created and entries for when the replication
was fixed after the HDFS restart. Here is the result of fsck:
$HADOOP_HOME/bin/hadoop fsck /
.......................................................
/data/hbase1/hregion_70236052/compaction.dir/hregion_70236052/info/
done: Under replicated blk_1984980330938654629. Target Replicas is 3
but found 1 replica(s).
..
/data/hbase1/hregion_70236052/info/info/2807320534360768620: Under
replicated blk_1717622121416314549. Target Replicas is 3 but found 1
replica(s).
.
/data/hbase1/hregion_70236052/info/mapfiles/2807320534360768620/
data: Under replicated blk_-5019714262388221150. Target Replicas is
3 but found 1 replica(s).
.
/data/hbase1/hregion_70236052/info/mapfiles/2807320534360768620/
index: Under replicated blk_4522585614366970680. Target Replicas is
3 but found 1 replica(s).
.........................................
/data/hbase1/log_10.100.11.63_1199307142676_60020/hlog.dat.000:
Under replicated blk_-2871471426720379908. Target Replicas is 3 but
found 1 replica(s).
.......
/data/hbase1/log_10.100.11.65_1199307142711_60020/hlog.dat.000:
MISSING 1 blocks of total size 0 B.
.Status: CORRUPT
Total size: 71009158262 B
Total blocks: 16318 (avg. block size 4351584 B)
Total dirs: 21416
Total files: 16253
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 5 (0.03064101 %)
Target replication factor: 3
Real replication factor: 2.9993873
The filesystem under path '/' is CORRUPT
-Chris
On Jan 4, 2008, at 1:02 PM, Raghu Angadi wrote:
This is of course not expected. A more detailed info or log message
would help. Do you know if there is at least one good block?
Sometimes, the remaining "good" block might actually be corrupted
and thus can not replicate itself. Restarting might just have
brought up the datanodes that were down (for whatever reason)
before the restart.
Raghu.
Chris Kline wrote:
fsck reports several under replicated blocks, but these do not get
fixed until I restart DFS. fsck also reports a missing block at
the same time, but this should affect the function of fixing under
replicated blocks. Has anyone seen this before?
I'm running 0.15.0.
-Chris Kline
-Chris
We're hiring engineers. $10,007 reward for referrals we hire.
