i check regionserver logs
What should I do, I only know a little bit HLog
2013-09-26 10:37:13,478 WARN org.apache.hadoop.hbase.util.FSHDFSUtils:
Cannot recoverLease after trying for 900000ms
(hbase.lease.recovery.timeout); continuing, but may be DATALOSS!!!;
attempt=16 on
file=hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting/hydra0006%2C60020%2C1379926437471.1380157500804
after 921109ms
2013-09-26 10:37:13,519 WARN org.apache.hadoop.hbase.regionserver.wal.HLog:
Lease should have recovered. This is not expected. Will retry
java.io.IOException: Cannot obtain block length for
LocatedBlock{BP-1087715125-192.5.1.50-1378889582109:blk_-8658284328699269340_21570;
getBlockSize()=0; corrupt=false; offset=0; locs=[192.5.1.56:50010,
192.5.1.52:50010, 192.5.1.55:50010]}
at
org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:319)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:263)
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:205)
at
org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:198)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1117)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:249)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:82)
at
org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1787)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.openFile(SequenceFileLogReader.java:62)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1707)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:713)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:846)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:759)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:371)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:115)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:283)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:214)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:182)
at java.lang.Thread.run(Thread.java:722)
2013-09-26 10:40:05,900 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=2.02 MB,
free=243.83 MB, max=245.84 MB, blocks=0, accesses=0, hits=0, hitRatio=0,
cachingAccesses=0, cachingHits=0, cachingHitsRatio=0, evictions=0,
evicted=0, evictedPerRun=NaN
2013-09-26 10:40:07,770 DEBUG
org.apache.hadoop.hbase.regionserver.LogRoller: Hlog roll period 3600000ms
elapsed
2013-09-26 10:42:14,404 ERROR
org.apache.hadoop.hbase.regionserver.wal.HLog: Can't open after 300
attempts and 300926ms for
hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting/hydra0006%2C60020%2C1379926437471.1380157500804
2013-09-26 10:42:14,404 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Processed 0 edits
across 0 regions threw away edits for 0 regions; log
file=hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting/hydra0006%2C60020%2C1379926437471.1380157500804
is corrupted = false progress failed = false
2013-09-26 10:42:14,404 WARN
org.apache.hadoop.hbase.regionserver.SplitLogWorker: log splitting of
hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting/hydra0006%2C60020%2C1379926437471.1380157500804
failed, returning error
java.io.IOException: Cannot obtain block length for
LocatedBlock{BP-1087715125-192.5.1.50-1378889582109:blk_-8658284328699269340_21570;
getBlockSize()=0; corrupt=false; offset=0; locs=[192.5.1.56:50010,
192.5.1.52:50010, 192.5.1.55:50010]}
at
org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:319)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:263)
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:205)
at
org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:198)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1117)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:249)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:82)
at
org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1787)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.openFile(SequenceFileLogReader.java:62)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1707)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:713)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:846)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:759)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:371)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:115)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:283)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:214)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:182)
at java.lang.Thread.run(Thread.java:722)
2013-09-26 10:42:14,418 INFO
org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully
transitioned task
/hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
to final state err
2013-09-26 10:42:14,419 INFO
org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker
hydra0006,60020,1380159605759 done with task
/hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
in 1222058ms
2013-09-26 10:42:14,427 DEBUG
org.apache.hadoop.hbase.regionserver.SplitLogWorker: tasks arrived or
departed
2013-09-26 10:42:44,450 DEBUG
org.apache.hadoop.hbase.regionserver.SplitLogWorker: tasks arrived or
departed
2013-09-26 10:42:44,497 INFO
org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker
hydra0006,60020,1380159605759 acquired task
/hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0003%2C60020%2C1379926447350-splitting%2Fhydra0003%252C60020%252C1379926447350.1380157487667
2013-09-26 10:42:44,499 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog:
hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667,
length=63861780
2013-09-26 10:42:44,506 INFO org.apache.hadoop.hbase.util.FSHDFSUtils:
Recovering lease on dfs file
hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667
2013-09-26 10:42:44,507 INFO org.apache.hadoop.hbase.util.FSHDFSUtils:
recoverLease=true, attempt=0 on
file=hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667
after 1ms
2013-09-26 10:42:44,512 WARN org.apache.hadoop.hdfs.DFSClient: Failed to
connect to /192.5.1.56:50010 for block, add to deadNodes and continue.
java.io.IOException: Got error for OP_READ_BLOCK, self=/192.5.1.56:37058,
remote=/192.5.1.56:50010, for file
/hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667,
for pool BP-1087715125-192.5.1.50-1378889582109 block
785178848034028699_21541
java.io.IOException: Got error for OP_READ_BLOCK, self=/192.5.1.56:37058,
remote=/192.5.1.56:50010, for file
/hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667,
for pool BP-1087715125-192.5.1.50-1378889582109 block
785178848034028699_21541
at
org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:429)
at
org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:394)
at
org.apache.hadoop.hdfs.BlockReaderFactory.newBlockReader(BlockReaderFactory.java:137)
at
org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:1103)
at
org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:538)
at
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:750)
at
org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:794)
at java.io.DataInputStream.read(DataInputStream.java:149)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at
org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1800)
at
org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:713)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:846)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:759)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:371)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:115)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:283)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:214)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:182)
at java.lang.Thread.run(Thread.java:722)
2013/9/26 Jimmy Xiang <[email protected]>
In the region server log, you should see the details about the failure.
On Wed, Sep 25, 2013 at 7:31 PM, kun yan <[email protected]> wrote:
I use importtsv import data into HDFS, but during the power outage.
Horrible, and then I re-import the data.（hbase 0.94）
the exception
org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find
region for data2,04684015,99999999999999 after 10 tries.
I look HMaster logs as follows:
2013-09-26 10:21:17,874 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
unassigned
= 0
2013-09-26 10:21:18,875 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
unassigned
= 0
2013-09-26 10:21:19,875 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
unassigned
= 0
2013-09-26 10:21:20,875 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
unassigned
= 0
2013-09-26 10:21:21,876 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
unassigned
= 0
2013-09-26 10:21:22,875 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
unassigned
= 0
2013-09-26 10:21:23,875 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
unassigned
= 0
2013-09-26 10:21:24,875 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
unassigned
= 0
2013-09-26 10:21:25,436 INFO
org.apache.hadoop.hbase.master.SplitLogManager: task
/hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
entered state err hydra0004,60020,1380159614688
2013-09-26 10:21:25,436 WARN
org.apache.hadoop.hbase.master.SplitLogManager: Error splitting
/hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
2013-09-26 10:21:25,436 WARN
org.apache.hadoop.hbase.master.SplitLogManager: error while splitting
logs
in
[hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting,
hdfs://hydra0001:8020/hbase/.logs/hydra0004,60020,1379926440171-splitting,
hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting]
installed = 2 but only 0 done
2013-09-26 10:21:25,436 WARN
org.apache.hadoop.hbase.master.MasterFileSystem: Failed splitting of
[hydra0003,60020,1379926447350, hydra0004,60020,1379926440171,
hydra0006,60020,1379926437471]
java.io.IOException: error or interrupted while splitting logs in
[hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting,
hdfs://hydra0001:8020/hbase/.logs/hydra0004,60020,1379926440171-splitting,
hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting]
Task = installed = 2 done = 0 error = 2
at
org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:282)
at
org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:300)
at
org.apache.hadoop.hbase.master.MasterFileSystem.splitLogAfterStartup(MasterFileSystem.java:242)
at
org.apache.hadoop.hbase.master.HMaster.splitLogAfterStartup(HMaster.java:661)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:580)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:396)
at java.lang.Thread.run(Thread.java:722)
--
In the Hadoop world, I am just a novice, explore the entire Hadoop
ecosystem, I hope one day I can contribute their own code
YanBit
[email protected]
--
In the Hadoop world, I am just a novice, explore the entire Hadoop
ecosystem, I hope one day I can contribute their own code
YanBit
[email protected]
