[
https://issues.apache.org/jira/browse/HBASE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
schubert zhang updated HBASE-1061:
----------------------------------
Attachment: hbase-stephen-regionserver-nd4-rack0-cloud.log
I enbaled the DEBUG log, and the log file is attached.
The state of HDFS is OK,
Node Last Contact Admin State Size (GB) Used (%) Used
(%) Remaining (GB) Blocks
nd1-rack0-cloud 1 In Service 822.8 1.13
755.27 806
nd2-rack0-cloud 2 In Service 822.8 1.1
755.4 3204
nd3-rack0-cloud 1 In Service 822.8 1.17
755.14 1918
nd4-rack0-cloud 2 In Service 822.8 1.18
755.13 2325
The loader code like following code:
HTable table = new HTable(ConfInstance.instance(), tableName);
for(int i = 0; ;++i) {
Date start = new Date(date.getTime() -
random.nextInt(30*24*60*60) * 1000L);
Integer end = random.nextInt(900000) + 100000;
String imsi = TestAdd.imsiBegin + end.toString();
BatchUpdate update = new BatchUpdate(imsi +
start.getTime());
update.put("gdr:start_time",
TypeDefine.toBytes(start.getTime()));
update.put("gdr:msisdn", TypeDefine.toBytes(imsi));
update.put("gdr:sourceip",
TypeDefine.toBytes(random.nextInt()));
update.put("gdr:destip",
TypeDefine.toBytes(random.nextInt()));
update.put("gdr:gdrtype",
TypeDefine.toBytes(random.nextInt(10) + 1));
update.put("gdr:req_num",
TypeDefine.toBytes(random.nextInt(10) + 1));
update.put("gdr:apn",
TypeDefine.toBytes(TestAdd.apns[random.nextInt(2)]));
update.put("gdr:gdrtime",
TypeDefine.toBytes(random.nextInt(10000) + 1));
update.put("gdr:result",
TypeDefine.toBytes(random.nextInt(2)));
update.put("gdr:gtpver",
TypeDefine.toBytes(random.nextInt(100) + 1));
update.put("gdr:remoteno",
TypeDefine.toBytes(random.nextInt(1000) + 1));
update.put("gdr:frontno",
TypeDefine.toBytes(random.nextInt(1000) + 1));
update.put("gdr:url",
TypeDefine.toBytes(TestAdd.urls[random.nextInt(TestAdd.urls.length)]));
try {
table.commit(update);
} catch (Exception e) {
e.printStackTrace();
Thread.sleep(60000);
}
> Region Server throw NotServingRegionException when batchUpdate during
> compaction/spliting and then shutdown
> ------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1061
> URL: https://issues.apache.org/jira/browse/HBASE-1061
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.18.1
> Environment: 1 master, 4 region server
> Hadoop 0.18.1
> When about 30GB 30,000,000 rows loaded
> Reporter: schubert zhang
> Attachments: hbase-stephen-regionserver-nd4-rack0-cloud.log
>
>
> 2008-12-13 21:30:58,184 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
> Rolling hlog. Number of entries: 30015
> 2008-12-13 21:30:58,595 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> starting compaction on region GDR,138884579261228161625885,1229166027688
> 2008-12-13 21:31:00,162 INFO org.apache.hadoop.hbase.regionserver.HLog: New
> log writer created at
> /hbase/log_10.24.1.12_1229150189049_60020/hlog.dat.1229175060158
> 2008-12-13 21:31:00,162 INFO org.apache.hadoop.hbase.regionserver.HLog:
> removing old log file
> /hbase/log_10.24.1.12_1229150189049_60020/hlog.dat.1229173244235 whose
> highest sequence/edit id is 145008256
> 2008-12-13 21:31:00,165 INFO org.apache.hadoop.hbase.regionserver.HLog:
> removing old log file
> /hbase/log_10.24.1.12_1229150189049_60020/hlog.dat.1229173248573 whose
> highest sequence/edit id is 145038271
> 2008-12-13 21:31:05,571 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> compaction completed on region GDR,138884579261228161625885,1229166027688 in
> 6sec
> 2008-12-13 21:31:05,571 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Starting split of region GDR,138884579261228161625885,1229166027688
> 2008-12-13 21:31:05,735 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> closed GDR,138884579261228161625885,1229166027688
> 2008-12-13 21:31:05,863 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 4 on 60020, call batchUpdate([...@4df611bd, row => 138884603141228853661885,
> {column => gdr:start_time, value => '...', column => gdr:imsi, value =>
> '...', column => gdr:msisdn, value => '...', column => gdr:sourceip, value =>
> '...', column => gdr:destip, value => '...', column => gdr:gdrtype, value =>
> '...', column => gdr:req_num, value => '...', column => gdr:apn, value =>
> '...', column => gdr:gdrtime, value => '...', column => gdr:result, value =>
> '...', column => gdr:gtpver, value => '...', column => gdr:remoteno, value =>
> '...', column => gdr:frontno, value => '...', column => gdr:offset, value =>
> '...', column => gdr:url, value => '...'}, -1) from 10.24.1.10:39000: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> GDR,138884579261228161625885,1229166027688 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region
> GDR,138884579261228161625885,1229166027688 closed
> at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1836)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1901)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1432)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1415)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-13 21:31:07,594 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> region GDR,138884579261228161625885,1229175065574/642186371 available
> 2008-12-13 21:31:07,594 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> closed GDR,138884579261228161625885,1229175065574
> 2008-12-13 21:31:07,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 9 on 60020, call batchUpdate([...@7792b8d, row => 138884603141228853661885,
> {column => gdr:start_time, value => '...', column => gdr:imsi, value =>
> '...', column => gdr:msisdn, value => '...', column => gdr:sourceip, value =>
> '...', column => gdr:destip, value => '...', column => gdr:gdrtype, value =>
> '...', column => gdr:req_num, value => '...', column => gdr:apn, value =>
> '...', column => gdr:gdrtime, value => '...', column => gdr:result, value =>
> '...', column => gdr:gtpver, value => '...', column => gdr:remoteno, value =>
> '...', column => gdr:frontno, value => '...', column => gdr:offset, value =>
> '...', column => gdr:url, value => '...'}, -1) from 10.24.1.10:39000: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> GDR,138884579261228161625885,1229166027688 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region
> GDR,138884579261228161625885,1229166027688 closed
> at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1836)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1901)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1432)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1415)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-13 21:31:08,346 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> region GDR,138884596811227665528885,1229175065574/1240335718 available
> The many region throw such exception. and then the region server shutdown
> 2008-12-13 22:01:46,700 WARN org.apache.hadoop.dfs.DFSClient: DataStreamer
> Exception: java.io.IOException: Unable to create new block.
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2349)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1735)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1912)
> 2008-12-13 22:01:46,701 WARN org.apache.hadoop.dfs.DFSClient: Error Recovery
> for block blk_-1991134958879829135_36755 bad datanode[1]
> 2008-12-13 22:01:46,701 FATAL org.apache.hadoop.hbase.regionserver.Flusher:
> Replay of hlog required. Forcing server shutdown
> and at last:
> 008-12-13 22:04:53,414 INFO org.apache.hadoop.ipc.Client: Retrying connect to
> server: /10.24.1.18:60020. Already tried 6 time(s).
> 2008-12-13 22:04:54,417 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /10.24.1.18:60020. Already tried 7 time(s).
> 2008-12-13 22:04:55,420 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /10.24.1.18:60020. Already tried 8 time(s).
> 2008-12-13 22:04:56,423 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /10.24.1.18:60020. Already tried 9 time(s).
> 2008-12-13 22:07:04,556 ERROR
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed
> for region GDR,138883452271227827193885,1229167157710
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.client.ServerCallable.getRegionName(ServerCallable.java:71)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
> at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
> at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.split(CompactSplitThread.java:167)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:85)
> 2008-12-13 22:07:04,558 INFO
> org.apache.hadoop.hbase.regionserver.CompactSplitThread:
> regionserver/0:0:0:0:0:0:0:0:60020.compactor exiting
> 2008-12-13 22:07:04,558 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> regionserver/0:0:0:0:0:0:0:0:60020 exiting
> 2008-12-13 22:07:04,627 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown thread.
> 2008-12-13 22:07:04,627 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread complete
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
