Zookeeper session got closed while trying to assign the region to RS using hbck
-fix
------------------------------------------------------------------------------------
Key: HBASE-5781
URL: https://issues.apache.org/jira/browse/HBASE-5781
Project: HBase
Issue Type: Bug
Components: hbck
Reporter: Kristam Subba Swathi
After running the hbck in the cluster ,it is found that one region is not
assigned
So the hbck -fix is used to fix this
But the assignment didnt happen since the zookeeper session is closed
Please find the attached trace for more details
-----------------------------------------
Trying to fix unassigned region...
12/04/03 11:02:57 INFO util.HBaseFsckRepair: Region still in transition,
waiting for it to become assigned: {NAME =>
'ufdr,002300,1333379123498.00871fbd7583512e12c4eb38e900be8d.', STARTKEY =>
'002300', ENDKEY => '002311', ENCODED => 00871fbd7583512e12c4eb38e900be8d,}
12/04/03 11:02:58 INFO client.HConnectionManager$HConnectionImplementation:
Closed zookeeper sessionid=0x236738a2630000a
12/04/03 11:02:58 INFO zookeeper.ZooKeeper: Session: 0x236738a2630000a closed
ERROR: Region { meta =>
ufdr,010444,1333379123857.01594219211d0035b9586f98954462e1., hdfs =>
hdfs://10.18.40.25:9000/hbase/ufdr/01594219211d0035b9586f98954462e1, deployed
=> } not deployed on any region server.
Trying to fix unassigned region...
12/04/03 11:02:58 INFO zookeeper.ClientCnxn: EventThread shut down
12/04/03 11:02:58 WARN zookeeper.ZKUtil: hconnection-0x236738a2630000a Unable
to set watcher on znode (/hbase)
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode =
Session expired for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626)
at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211)
at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325)
at
org.apache.hadoop.hbase.util.HBaseFsckRepair.forceOfflineInZK(HBaseFsckRepair.java:109)
at
org.apache.hadoop.hbase.util.HBaseFsckRepair.fixUnassigned(HBaseFsckRepair.java:92)
at
org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1235)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1351)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1114)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:356)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:375)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2894)
12/04/03 11:02:58 ERROR zookeeper.ZooKeeperWatcher:
hconnection-0x236738a2630000a Received unexpected KeeperException, re-throwing
exception
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode =
Session expired for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626)
at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211)
at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325)
at
org.apache.hadoop.hbase.util.HBaseFsckRepair.forceOfflineInZK(HBaseFsckRepair.java:109)
at
org.apache.hadoop.hbase.util.HBaseFsckRepair.fixUnassigned(HBaseFsckRepair.java:92)
at
org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1235)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1351)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1114)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:356)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:375)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2894)
12/04/03 11:02:58 INFO client.HConnectionManager$HConnectionImplementation:
This client just lost it's session with ZooKeeper, trying to reconnect.
12/04/03 11:02:58 INFO client.HConnectionManager$HConnectionImplementation:
Trying to reconnect to zookeeper
12/04/03 11:02:58 INFO zookeeper.ZooKeeper: Initiating client connection,
connectString=10.18.40.21:2181,10.18.40.25:2181,10.18.40.93:2181
sessionTimeout=60000 watcher=hconnection
12/04/03 11:02:58 INFO zookeeper.ClientCnxn: Opening socket connection to
server /10.18.40.93:2181
12/04/03 11:02:58 INFO zookeeper.RecoverableZooKeeper: The identifier of this
process is 18333@HOST-10-18-40-93
12/04/03 11:02:58 WARN client.ZooKeeperSaslClient: SecurityException:
java.lang.SecurityException: Unable to locate a login configuration occurred
when trying to find JAAS configuration.
12/04/03 11:02:58 INFO client.ZooKeeperSaslClient: Client will not
SASL-authenticate because the default JAAS configuration section 'Client' could
not be found. If you are not using SASL, you may ignore this. On the other
hand, if you expected SASL to work, please fix your JAAS configuration.
12/04/03 11:02:58 INFO zookeeper.ClientCnxn: Socket connection established to
HOST-10-18-40-93/10.18.40.93:2181, initiating session
12/04/03 11:02:58 INFO zookeeper.ClientCnxn: Session establishment complete on
server HOST-10-18-40-93/10.18.40.93:2181, sessionid = 0x3367392d5140018,
negotiated timeout = 40000
12/04/03 11:02:58 INFO client.HConnectionManager$HConnectionImplementation:
Reconnected successfully. This disconnect could have been caused by a network
partition or a long-running GC pause, either way it's recommended that you
verify your environment.
Exception in thread "main" org.apache.hadoop.hbase.MasterNotRunningException
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:686)
at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211)
at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325)
at
org.apache.hadoop.hbase.util.HBaseFsckRepair.forceOfflineInZK(HBaseFsckRepair.java:109)
at
org.apache.hadoop.hbase.util.HBaseFsckRepair.fixUnassigned(HBaseFsckRepair.java:92)
at
org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1235)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1351)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1114)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:356)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:375)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2894)
Please find the attached file for more details..
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
