Which hbase release do you use ? Have you checked region server log to see why log splitting had issues ?
Cheers On Jun 26, 2014, at 1:55 AM, "lilibiao2014" <[email protected]> wrote: > Hey guys, > > Yesterday our Hbase cluster had 4 of 11 regionserver don't work well, that > the numberOfOnlineRegions= 0 . > And when we restart the cluster,not only 4 but all of our regionservers this > occurs. > Here is the hbase master's log.Except the exception of the log ,we also find > few zookeeper's exception and log splitting exception.We can't find the real > cause. > > Hope that helps and forgive my poor English : ) > Thanks > Lee > > 2014-06-26 16:00:20,220 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:21,220 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:22,220 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:23,222 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:24,222 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:25,224 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:26,224 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:27,225 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:28,227 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:29,227 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:30,229 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:31,231 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:32,231 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:33,233 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:34,234 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:35,235 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:36,235 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:37,235 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:38,237 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:39,238 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:40,238 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:41,240 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:42,240 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:43,241 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:44,243 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:45,244 INFO org.apache.hadoop.hbase.master.SplitLogManager: > Skipping resubmissions of task > /hbase/splitlog/hdfs%3A%2F%2Fjyw-o-hadoop00%3A9000%2Fhbase%2F.logs%2Fjyw-o-h > adoop05.light.soufun.com%2C60020%2C1403110267692-splitting%2Fjyw-o-hadoop05. > light.soufun.com%252C60020%252C1403110267692.1403720357924 because threshold > 3 reached > 2014-06-26 16:00:45,244 INFO org.apache.hadoop.hbase.master.SplitLogManager: > Skipping resubmissions of task > /hbase/splitlog/hdfs%3A%2F%2Fjyw-o-hadoop00%3A9000%2Fhbase%2F.logs%2Fjyw-o-h > adoop05.light.soufun.com%2C60020%2C1403110267692-splitting%2Fjyw-o-hadoop05. > light.soufun.com%252C60020%252C1403110267692.1403723402640 because threshold > 3 reached > 2014-06-26 16:00:45,244 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:46,245 DEBUG > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = > 0 > 2014-06-26 16:00:46,972 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server > listener on 60000: readAndProcess threw exception java.io.IOException: > Connection reset by peer. Count of bytes read: 0 > java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcher.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198) > at sun.nio.ch.IOUtil.read(IOUtil.java:171) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245) > at > org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1796) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServe > r.java:1179) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748 > ) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServe > r.java:539) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java > :514) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja > va:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 > 08) > at java.lang.Thread.run(Thread.java:662) > >
