Vamshi, max value for hbase.hregion.max.filesize to 10MB seems too small. Did you mean 10GB?
Regards, Dhaval ________________________________ From: Vamshi Krishna <[email protected]> To: [email protected]; zhoushuaifeng <[email protected]> Sent: Friday, 23 August 2013 9:38 AM Subject: Re: Will hbase automatically distribute the data across region servers or NOT..?? Thanks for the clarifications. I am using hbase-0.94.10 and zookeepr-3.4.5 But I am running into different issues . I set hbase.hregion.max.filesize to 10Mb and i am inserting 10 million rows in to hbase table. During the insertion after some time, suddenly master is going down. I don't know what is the reason for such peculiar behavior. I found in master log below content and not able to make out what exactly the mistake. Please somebody help. master-log: 2013-08-23 18:56:36,865 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: [] 2013-08-23 18:56:36,866 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f. state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273 .. Cannot transit it to OFFLINE. java.lang.IllegalStateException: Unexpected state : scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f. state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273 .. Cannot transit it to OFFLINE. at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394) at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2013-08-23 18:56:36,867 INFO org.apache.hadoop.hbase.master.HMaster: Aborting 2013-08-23 18:56:36,867 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60000 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60000: exiting 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60000: exiting 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60000: exiting 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60000 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC Server handler 2 on 60000: exiting 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC Server handler 1 on 60000: exiting 2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster$2: vamshi,60000,1377263788019-BalancerChore exiting 2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster: Stopping infoServer 2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.cleaner.HFileCleaner: master-vamshi,60000,1377263788019.archivedHFileCleaner exiting 2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.CatalogJanitor: vamshi,60000,1377263788019-CatalogJanitor exiting 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC Server handler 0 on 60000: exiting 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60000: exiting 2013-08-23 18:56:36,874 INFO org.mortbay.log: Stopped [email protected]:60010 2013-08-23 18:56:36,874 INFO org.apache.hadoop.hbase.master.cleaner.LogCleaner: master-vamshi,60000,1377263788019.oldLogCleaner exiting 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60000: exiting 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60000: exiting 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60000: exiting 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60000: exiting 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder 2013-08-23 18:56:36,876 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60000: exiting 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000: exiting 2013-08-23 18:56:36,877 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x140ab519b0f0000 Unable to set watcher on znode (/hbase/unassigned/05e30711673614f6b41a364c76f3f05f) java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:450) at org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:271) at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394) at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2013-08-23 18:56:36,876 WARN org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force node into OFFLINE state before completing assignment but failed to do so for scores,\x00\x00\x00\x00\x00\x08b8,1377264147374.39794b7deea3203fc260756f5038d6f8. state=OFFLINE, ts=1377264396802, server=null 2013-08-23 18:56:36,876 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x140ab519b0f0000 Unable to get data of znode /hbase/unassigned/d476f8442ce31de90b60080b74daf47f java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1149) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:290) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:746) at org.apache.hadoop.hbase.zookeeper.ZKAssign.getDataNoWatch(ZKAssign.java:904) at org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:283) at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394) at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 2013-08-23 18:56:36,877 WARN org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force node into OFFLINE state before completing assignment but failed to do so for scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f. state=OFFLINE, ts=1377264396862, server=null 2013-08-23 18:56:36,877 WARN org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force node into OFFLINE state before completing assignment but failed to do so for scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f. state=OFFLINE, ts=1377264396813, server=null 2013-08-23 18:56:36,882 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_FAILED_OPEN, server=vamshi_RS,60020,1377263792053, region=d476f8442ce31de90b60080b74daf47f 2013-08-23 18:56:36,882 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f. destination server is vamshi,60020,1377263789273 2013-08-23 18:56:36,882 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f. so generated a random one; hri=scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f., src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1) available servers 2013-08-23 18:56:36,882 ERROR org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit [ClosedRegionHandler-vamshi,60000,1377263788019-38] because the executor is missing. Is this process shutting down? 2013-08-23 18:56:36,906 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@451415c8 2013-08-23 18:56:36,906 INFO org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor: vamshi,60000,1377263788019.timeoutMonitor exiting 2013-08-23 18:56:36,906 INFO org.apache.hadoop.hbase.master.AssignmentManager$TimerUpdater: vamshi,60000,1377263788019.timerUpdater exiting 2013-08-23 18:56:36,907 INFO org.apache.hadoop.hbase.master.SplitLogManager$TimeoutMonitor: vamshi,60000,1377263788019.splitLogManagerTimeoutMonitor exiting 2013-08-23 18:56:36,910 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_FAILED_OPEN, server=vamshi_RS,60020,1377263792053, region=05e30711673614f6b41a364c76f3f05f 2013-08-23 18:56:36,911 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f. destination server is vamshi,60020,1377263789273 2013-08-23 18:56:36,911 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f. so generated a random one; hri=scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f., src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1) available servers 2013-08-23 18:56:36,911 ERROR org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit [ClosedRegionHandler-vamshi,60000,1377263788019-39] because the executor is missing. Is this process shutting down? 2013-08-23 18:56:36,912 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/unassigned/d476f8442ce31de90b60080b74daf47f 2013-08-23 18:56:36,912 INFO org.apache.hadoop.hbase.util.RetryCounter: Sleeping 2000ms before retry #1... 2013-08-23 18:56:36,914 INFO org.apache.zookeeper.ZooKeeper: Session: 0x140ab519b0f0000 closed 2013-08-23 18:56:36,914 INFO org.apache.hadoop.hbase.master.HMaster: HMaster main thread exiting 2013-08-23 18:56:36,914 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master java.lang.RuntimeException: HMaster Aborted at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:160) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2100) My hbase-site.xml : <configuration> <property> <name>hbase.rootdir</name> <value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/hbstmp/</value> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.master</name> <value>vamshi</value> </property> <property> <name>hbase.zookeeper.property.clientPort</name> <value>2181</value> </property> <property> <name>hbase.hregion.max.filesize</name> <value>10485760</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>vamshi</value> </property> <property> <name>hbase.zookeeper.property.dataDir</name> <value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/zkptmp</value> </property> <property> <name>hbase.zookeeper.property.maxClientCnxns</name> <value>1024</value> </property> <property> <name>hbase.coprocessor.user.region.classes</name> <value>com.bil.coproc.ColumnAggregationEndpoint</value> </property> </configuration> On Fri, Aug 23, 2013 at 7:00 PM, Frank Chow <[email protected]> wrote: > Hi, > You may should check if the compact is on. If data size in a region is max > than the limition, region will split and balance after a major > compaction(Usually occur automatically). > You can manually by run the compact operaction by the shell commond: > compact <tableName>, or major_compact <tableName> > > > > > Frank Chow -- *Regards* * Vamshi Krishna *
