The HBase version we are using is CDH3U0. Thanks Weihua
2011/7/4 Weihua JIANG <[email protected]>: > Hi all, > > We encountered a problem about region not onlining. A region is > splitted by a closing RS and then this RS down. It seems master has > known this split but it doesn't tried to make it online. Log from > master > 2011-06-30 22:58:52,945 DEBUG > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined > and split region > CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.; > checking daughter presence > 2011-06-30 22:58:52,946 DEBUG > org.apache.hadoop.hbase.master.AssignmentManager: Handling > transition=RS_ZK_REGION_OPENING, > server=hadoop01.sh.intel.com,50820,1309421825940, > region=ed60ec735e30db1d99290995eb1cd2d7 > 2011-06-30 22:58:53,005 DEBUG > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter > CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e. > present > 2011-06-30 22:58:53,065 DEBUG > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter > CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294. > present > > Log from RS is: > 2011-06-30 22:57:05,207 WARN org.apache.hadoop.ipc.HBaseServer: IPC > Server handler 73 on 50820 caught: > java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324) > at > org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1342) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083) > > 2011-06-30 22:57:05,207 INFO org.apache.hadoop.ipc.HBaseServer: IPC > Server handler 73 on 50820: exiting > 2011-06-30 22:57:05,767 INFO > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closing > leases > 2011-06-30 22:57:05,768 INFO > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closed > leases > 2011-06-30 22:57:05,768 INFO > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > Closed zookeeper sessionid=0x130ba69074900b4 > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ZooKeeper: Session: > 0x130ba69074900b4 closed > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ClientCnxn: > EventThread shut down > 2011-06-30 22:57:05,857 DEBUG > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated > CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e. > 2011-06-30 22:57:05,863 DEBUG > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated > CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294. > 2011-06-30 22:57:05,911 INFO > org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region > CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd. > in META > 2011-06-30 22:57:05,942 INFO > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter > CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e. > in region .META.,,1, serverInfo=null > 2011-06-30 22:57:05,943 INFO > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening > daughter > CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e. > because stopping=false, stopped=true > 2011-06-30 22:57:05,950 INFO > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter > CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294. > in region .META.,,1, serverInfo=null > 2011-06-30 22:57:05,950 INFO > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening > daughter > CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294. > because stopping=false, stopped=true > 2011-06-30 22:57:06,004 INFO > org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META > updated, and report to master. > Parent=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd., > new regions: > CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e., > CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.. > Split took 1mins, 12sec > 2011-06-30 22:57:06,004 DEBUG > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for > Split Thread to finish... > 2011-06-30 22:57:06,004 DEBUG > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for > Large Compaction Thread to finish... > 2011-06-30 22:57:06,004 DEBUG > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for > Small Compaction Thread to finish... > 2011-06-30 22:57:06,004 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver50820 > exiting > 2011-06-30 22:57:06,090 INFO > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook > starting; hbase.shutdown.hook=true; > fsShutdownHook=Thread[Thread-15,5,main] > 2011-06-30 22:57:06,090 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown > hook > 2011-06-30 22:57:06,090 INFO > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs > shutdown hook thread. > 2011-06-30 22:57:06,196 INFO > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook > finished. > > > Thanks > Weihua >
