[
https://issues.apache.org/jira/browse/HBASE-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707012#action_12707012
]
stack commented on HBASE-1311:
------------------------------
Here is what I saw on master side:
{code}
2009-05-07 07:40:05,597 [RegionManager.rootScanner-EventThread] DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Got ZooKeeper
event, state: SyncConnected, type: None, path: null
2009-05-07 07:40:07,555 [RegionManager.rootScanner] INFO
org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of 1
row(s) of meta region {regionname: -ROOT-,,0, startKey: <>, server:
208.76.44.142:60021} complete
2009-05-07 07:40:19,983 [IPC Server handler 6 on 60000] DEBUG
org.apache.hadoop.hbase.master.ServerManager: Total Load: 1399, Num Servers: 3,
Avg Load: 467.0
2009-05-07 07:40:27,089 [main-EventThread] INFO
org.apache.hadoop.hbase.master.ServerManager:
aa0-000-15.u.powerset.com_1241637114572_60021 znode expired
2009-05-07 07:40:27,120 [HMaster] DEBUG org.apache.hadoop.hbase.master.HMaster:
Processing todo: ProcessServerShutdown of
aa0-000-15.u.powerset.com_1241637114572_60021
2009-05-07 07:40:27,120 [HMaster] INFO
org.apache.hadoop.hbase.master.RegionServerOperation: process shutdown of
server aa0-000-15.u.powerset.com_1241637114572_60021: logSplit: false,
rootRescanned: false, numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
2009-05-07 07:40:27,125 [HMaster] INFO
org.apache.hadoop.hbase.regionserver.HLog: Splitting 3 log(s) in
hdfs://aa0-000-12.u.powerset.com:9000/hbasetrunk2/@LOGS@/aa0-000-15.u.powerset.com_1241637114572_60021
2009-05-07 07:40:27,126 [HMaster] DEBUG
org.apache.hadoop.hbase.regionserver.HLog: Splitting 1 of 3:
hdfs://aa0-000-12.u.powerset.com:9000/hbasetrunk2/@LOGS@/aa0-000-15.u.powerset.com_1241637114572_60021/hlog.dat.1241676102482
2009-05-07 07:40:30,022 [main-EventThread] INFO
org.apache.hadoop.hbase.master.ServerManager:
aa0-000-14.u.powerset.com_1241637114470_60021 znode expired
2009-05-07 07:40:34,779 [IPC Server handler 7 on 60000] INFO
org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60000, call
regionServerReport(address: 208.76.44.141:60021, startcode: 1241637114470,
load: (requests=0, regions=466, usedHeap=861, maxHeap=1255),
[Lorg.apache.hadoop.hbase.HMsg;@191ea7e5,
[Lorg.apache.hadoop.hbase.HRegionInfo;@2ba0b845) from 208.76.44.141:5009
4: error: org.apache.hadoop.hbase.Leases$LeaseStillHeldException
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at
org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:201)
at
org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:604)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:642)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:911)
2009-05-07 07:40:34,792 [IPC Server handler 4 on 60000] INFO
org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000, call
regionServerReport(address: 208.76.44.141:60021, startcode: 1241637114470,
load: (requests=0, regions=466, usedHeap=861, maxHeap=1255),
[Lorg.apache.hadoop.hbase.HMsg;@5b7836c8,
[Lorg.apache.hadoop.hbase.HRegionInfo;@3154b362) from 208.76.44.141:5019
7: error: org.apache.hadoop.hbase.Leases$LeaseStillHeldException
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at
org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:201)
at
org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:604)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:642)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:911)
2009-05-07 07:40:34,801 [IPC Server handler 9 on 60000] INFO
org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60000, call
regionServerReport(address: 208.76.44.141:60021, startcode: 1241637114470,
load: (requests=0, regions=466, usedHeap=862, maxHeap=1255),
[Lorg.apache.hadoop.hbase.HMsg;@2fcd003b,
[Lorg.apache.hadoop.hbase.HRegionInfo;@3bca3a01) from 208.76.44.141:5019
7: error: org.apache.hadoop.hbase.Leases$LeaseStillHeldException
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at
org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:201)
at
org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:604)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:642)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:911)
{code}
Above repeats over and over.
> ZooKeeperWrapper: Failed to set watcher on ZNode /hbase/master
> --------------------------------------------------------------
>
> Key: HBASE-1311
> URL: https://issues.apache.org/jira/browse/HBASE-1311
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: Andrew Purtell
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
> Attachments: dumb-wrapper-hack.patch
>
>
> After about 12 hours of operation, this repeats over and over in the
> regionserver log:
> 2009-04-05 19:44:38,445 WARN
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to set watcher on
> ZNode /hbase/master
> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode
> = Session expired
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:709)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.watchMasterAddress(ZooKeeperWrapper.java:235)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.watchMasterAddress(HRegionServer.java:343)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.process(HRegionServer.java:339)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:366)
> 2009-04-05 19:44:38,445 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to set watcher on
> ZooKeeper master address. Retrying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
