Thanks for the clarifications.
I am using hbase-0.94.10 and zookeepr-3.4.5
But I am running into different issues .
I set hbase.hregion.max.filesize to 10Mb and i am inserting 10 million
rows in to hbase table. During the insertion after some time, suddenly
master is going down. I don't know what is the reason for such peculiar
behavior.
I found in master log below content and not able to make out what exactly
the mistake. Please somebody help.
master-log:
2013-08-23 18:56:36,865 FATAL org.apache.hadoop.hbase.master.HMaster:
Master server abort: loaded coprocessors are: []
2013-08-23 18:56:36,866 FATAL org.apache.hadoop.hbase.master.HMaster:
Unexpected state :
scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273 ..
Cannot transit it to OFFLINE.
java.lang.IllegalStateException: Unexpected state :
scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273 ..
Cannot transit it to OFFLINE.
at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
2013-08-23 18:56:36,867 INFO org.apache.hadoop.hbase.master.HMaster:
Aborting
2013-08-23 18:56:36,867 DEBUG org.apache.hadoop.hbase.master.HMaster:
Stopping service threads
2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60000
2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60000: exiting
2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60000: exiting
2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server listener on 60000
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
Server handler 2 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
Server handler 1 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster$2:
vamshi,60000,1377263788019-BalancerChore exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster:
Stopping infoServer
2013-08-23 18:56:36,873 INFO
org.apache.hadoop.hbase.master.cleaner.HFileCleaner:
master-vamshi,60000,1377263788019.archivedHFileCleaner exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.CatalogJanitor:
vamshi,60000,1377263788019-CatalogJanitor exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
Server handler 0 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.mortbay.log: Stopped
[email protected]:60010
2013-08-23 18:56:36,874 INFO
org.apache.hadoop.hbase.master.cleaner.LogCleaner:
master-vamshi,60000,1377263788019.oldLogCleaner exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server Responder
2013-08-23 18:56:36,876 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server Responder
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60000: exiting
2013-08-23 18:56:36,877 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
master:60000-0x140ab519b0f0000 Unable to set watcher on znode
(/hbase/unassigned/05e30711673614f6b41a364c76f3f05f)
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:450)
at
org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:271)
at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
2013-08-23 18:56:36,876 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
node into OFFLINE state before completing assignment but failed to do so
for
scores,\x00\x00\x00\x00\x00\x08b8,1377264147374.39794b7deea3203fc260756f5038d6f8.
state=OFFLINE, ts=1377264396802, server=null
2013-08-23 18:56:36,876 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
master:60000-0x140ab519b0f0000 Unable to get data of znode
/hbase/unassigned/d476f8442ce31de90b60080b74daf47f
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1149)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:290)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:746)
at
org.apache.hadoop.hbase.zookeeper.ZKAssign.getDataNoWatch(ZKAssign.java:904)
at
org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:283)
at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
2013-08-23 18:56:36,877 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
node into OFFLINE state before completing assignment but failed to do so
for
scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
state=OFFLINE, ts=1377264396862, server=null
2013-08-23 18:56:36,877 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
node into OFFLINE state before completing assignment but failed to do so
for
scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
state=OFFLINE, ts=1377264396813, server=null
2013-08-23 18:56:36,882 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_FAILED_OPEN, server=vamshi_RS,60020,1377263792053,
region=d476f8442ce31de90b60080b74daf47f
2013-08-23 18:56:36,882 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
for
scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
destination server is vamshi,60020,1377263789273
2013-08-23 18:56:36,882 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
plan was found (or we are ignoring an existing plan) for
scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
so generated a random one;
hri=scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.,
src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1) available
servers
2013-08-23 18:56:36,882 ERROR
org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
[ClosedRegionHandler-vamshi,60000,1377263788019-38] because the executor is
missing. Is this process shutting down?
2013-08-23 18:56:36,906 DEBUG
org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker
org.apache.hadoop.hbase.catalog.CatalogTracker@451415c8
2013-08-23 18:56:36,906 INFO
org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor:
vamshi,60000,1377263788019.timeoutMonitor exiting
2013-08-23 18:56:36,906 INFO
org.apache.hadoop.hbase.master.AssignmentManager$TimerUpdater:
vamshi,60000,1377263788019.timerUpdater exiting
2013-08-23 18:56:36,907 INFO
org.apache.hadoop.hbase.master.SplitLogManager$TimeoutMonitor:
vamshi,60000,1377263788019.splitLogManagerTimeoutMonitor exiting
2013-08-23 18:56:36,910 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_FAILED_OPEN, server=vamshi_RS,60020,1377263792053,
region=05e30711673614f6b41a364c76f3f05f
2013-08-23 18:56:36,911 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
for
scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
destination server is vamshi,60020,1377263789273
2013-08-23 18:56:36,911 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
plan was found (or we are ignoring an existing plan) for
scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
so generated a random one;
hri=scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.,
src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1) available
servers
2013-08-23 18:56:36,911 ERROR
org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
[ClosedRegionHandler-vamshi,60000,1377263788019-39] because the executor is
missing. Is this process shutting down?
2013-08-23 18:56:36,912 WARN
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for
/hbase/unassigned/d476f8442ce31de90b60080b74daf47f
2013-08-23 18:56:36,912 INFO org.apache.hadoop.hbase.util.RetryCounter:
Sleeping 2000ms before retry #1...
2013-08-23 18:56:36,914 INFO org.apache.zookeeper.ZooKeeper: Session:
0x140ab519b0f0000 closed
2013-08-23 18:56:36,914 INFO org.apache.hadoop.hbase.master.HMaster:
HMaster main thread exiting
2013-08-23 18:56:36,914 ERROR
org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: HMaster Aborted
at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:160)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2100)
My hbase-site.xml :
<configuration>
<property>
<name>hbase.rootdir</name>
<value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/hbstmp/</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.master</name>
<value>vamshi</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.hregion.max.filesize</name>
<value>10485760</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>vamshi</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/zkptmp</value>
</property>
<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value>1024</value>
</property>
<property>
<name>hbase.coprocessor.user.region.classes</name>
<value>com.bil.coproc.ColumnAggregationEndpoint</value>
</property>
</configuration>
On Fri, Aug 23, 2013 at 7:00 PM, Frank Chow <[email protected]> wrote:
> Hi,
> You may should check if the compact is on. If data size in a region is max
> than the limition, region will split and balance after a major
> compaction(Usually occur automatically).
> You can manually by run the compact operaction by the shell commond:
> compact <tableName>, or major_compact <tableName>
>
>
>
>
> Frank Chow
--
*Regards*
*
Vamshi Krishna
*
