[
https://issues.apache.org/jira/browse/HBASE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656330#action_12656330
]
stack commented on HBASE-1061:
------------------------------
Please enable DEBUG and repost (See FAQ for how). What are the stats on your
hdfs? Tell us about your uploader? Looks like your cluster may be suffering
overloading as seemed apparent from the other JIRA you opened.
> Region Server throw NotServingRegionException when batchUpdate during
> compaction/spliting and then shutdown
> ------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1061
> URL: https://issues.apache.org/jira/browse/HBASE-1061
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.18.1
> Environment: 1 master, 4 region server
> Hadoop 0.18.1
> When about 30GB 30,000,000 rows loaded
> Reporter: schubert zhang
>
> 2008-12-13 21:30:58,184 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
> Rolling hlog. Number of entries: 30015
> 2008-12-13 21:30:58,595 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> starting compaction on region GDR,138884579261228161625885,1229166027688
> 2008-12-13 21:31:00,162 INFO org.apache.hadoop.hbase.regionserver.HLog: New
> log writer created at
> /hbase/log_10.24.1.12_1229150189049_60020/hlog.dat.1229175060158
> 2008-12-13 21:31:00,162 INFO org.apache.hadoop.hbase.regionserver.HLog:
> removing old log file
> /hbase/log_10.24.1.12_1229150189049_60020/hlog.dat.1229173244235 whose
> highest sequence/edit id is 145008256
> 2008-12-13 21:31:00,165 INFO org.apache.hadoop.hbase.regionserver.HLog:
> removing old log file
> /hbase/log_10.24.1.12_1229150189049_60020/hlog.dat.1229173248573 whose
> highest sequence/edit id is 145038271
> 2008-12-13 21:31:05,571 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> compaction completed on region GDR,138884579261228161625885,1229166027688 in
> 6sec
> 2008-12-13 21:31:05,571 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Starting split of region GDR,138884579261228161625885,1229166027688
> 2008-12-13 21:31:05,735 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> closed GDR,138884579261228161625885,1229166027688
> 2008-12-13 21:31:05,863 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 4 on 60020, call batchUpdate([...@4df611bd, row => 138884603141228853661885,
> {column => gdr:start_time, value => '...', column => gdr:imsi, value =>
> '...', column => gdr:msisdn, value => '...', column => gdr:sourceip, value =>
> '...', column => gdr:destip, value => '...', column => gdr:gdrtype, value =>
> '...', column => gdr:req_num, value => '...', column => gdr:apn, value =>
> '...', column => gdr:gdrtime, value => '...', column => gdr:result, value =>
> '...', column => gdr:gtpver, value => '...', column => gdr:remoteno, value =>
> '...', column => gdr:frontno, value => '...', column => gdr:offset, value =>
> '...', column => gdr:url, value => '...'}, -1) from 10.24.1.10:39000: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> GDR,138884579261228161625885,1229166027688 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region
> GDR,138884579261228161625885,1229166027688 closed
> at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1836)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1901)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1432)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1415)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-13 21:31:07,594 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> region GDR,138884579261228161625885,1229175065574/642186371 available
> 2008-12-13 21:31:07,594 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> closed GDR,138884579261228161625885,1229175065574
> 2008-12-13 21:31:07,885 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 9 on 60020, call batchUpdate([...@7792b8d, row => 138884603141228853661885,
> {column => gdr:start_time, value => '...', column => gdr:imsi, value =>
> '...', column => gdr:msisdn, value => '...', column => gdr:sourceip, value =>
> '...', column => gdr:destip, value => '...', column => gdr:gdrtype, value =>
> '...', column => gdr:req_num, value => '...', column => gdr:apn, value =>
> '...', column => gdr:gdrtime, value => '...', column => gdr:result, value =>
> '...', column => gdr:gtpver, value => '...', column => gdr:remoteno, value =>
> '...', column => gdr:frontno, value => '...', column => gdr:offset, value =>
> '...', column => gdr:url, value => '...'}, -1) from 10.24.1.10:39000: error:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> GDR,138884579261228161625885,1229166027688 closed
> org.apache.hadoop.hbase.NotServingRegionException: Region
> GDR,138884579261228161625885,1229166027688 closed
> at
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1836)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1901)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1432)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1415)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> 2008-12-13 21:31:08,346 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> region GDR,138884596811227665528885,1229175065574/1240335718 available
> The many region throw such exception. and then the region server shutdown
> 2008-12-13 22:01:46,700 WARN org.apache.hadoop.dfs.DFSClient: DataStreamer
> Exception: java.io.IOException: Unable to create new block.
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2349)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1735)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1912)
> 2008-12-13 22:01:46,701 WARN org.apache.hadoop.dfs.DFSClient: Error Recovery
> for block blk_-1991134958879829135_36755 bad datanode[1]
> 2008-12-13 22:01:46,701 FATAL org.apache.hadoop.hbase.regionserver.Flusher:
> Replay of hlog required. Forcing server shutdown
> and at last:
> 008-12-13 22:04:53,414 INFO org.apache.hadoop.ipc.Client: Retrying connect to
> server: /10.24.1.18:60020. Already tried 6 time(s).
> 2008-12-13 22:04:54,417 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /10.24.1.18:60020. Already tried 7 time(s).
> 2008-12-13 22:04:55,420 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /10.24.1.18:60020. Already tried 8 time(s).
> 2008-12-13 22:04:56,423 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /10.24.1.18:60020. Already tried 9 time(s).
> 2008-12-13 22:07:04,556 ERROR
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed
> for region GDR,138883452271227827193885,1229167157710
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.client.ServerCallable.getRegionName(ServerCallable.java:71)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
> at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
> at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.split(CompactSplitThread.java:167)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:85)
> 2008-12-13 22:07:04,558 INFO
> org.apache.hadoop.hbase.regionserver.CompactSplitThread:
> regionserver/0:0:0:0:0:0:0:0:60020.compactor exiting
> 2008-12-13 22:07:04,558 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> regionserver/0:0:0:0:0:0:0:0:60020 exiting
> 2008-12-13 22:07:04,627 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown thread.
> 2008-12-13 22:07:04,627 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread complete
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
