[jira] [Updated] (HBASE-5781) Zookeeper session got closed while trying to assign the region to RS using hbck -fix
[ https://issues.apache.org/jira/browse/HBASE-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5781: - Fix Version/s: (was: 0.95.0) 0.92.2 Zookeeper session got closed while trying to assign the region to RS using hbck -fix Key: HBASE-5781 URL: https://issues.apache.org/jira/browse/HBASE-5781 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.90.7, 0.92.1, 0.94.0, 0.95.2 Reporter: Kristam Subba Swathi Assignee: Jonathan Hsieh Priority: Critical Fix For: 0.92.2, 0.94.0 Attachments: hbase-5781.patch After running the hbck in the cluster ,it is found that one region is not assigned So the hbck -fix is used to fix this But the assignment didnt happen since the zookeeper session is closed Please find the attached trace for more details - Trying to fix unassigned region... 12/04/03 11:02:57 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {NAME = 'ufdr,002300,179123498.00871fbd7583512e12c4eb38e900be8d.', STARTKEY = '002300', ENDKEY = '002311', ENCODED = 00871fbd7583512e12c4eb38e900be8d,} 12/04/03 11:02:58 INFO client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x236738a263a 12/04/03 11:02:58 INFO zookeeper.ZooKeeper: Session: 0x236738a263a closed ERROR: Region { meta = ufdr,010444,179123857.01594219211d0035b9586f98954462e1., hdfs = hdfs://10.18.40.25:9000/hbase/ufdr/01594219211d0035b9586f98954462e1, deployed = } not deployed on any region server. Trying to fix unassigned region... 12/04/03 11:02:58 INFO zookeeper.ClientCnxn: EventThread shut down 12/04/03 11:02:58 WARN zookeeper.ZKUtil: hconnection-0x236738a263a Unable to set watcher on znode (/hbase) org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626) at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211) at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325) at org.apache.hadoop.hbase.util.HBaseFsckRepair.forceOfflineInZK(HBaseFsckRepair.java:109) at org.apache.hadoop.hbase.util.HBaseFsckRepair.fixUnassigned(HBaseFsckRepair.java:92) at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1235) at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1351) at org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1114) at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:356) at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:375) at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2894) 12/04/03 11:02:58 ERROR zookeeper.ZooKeeperWatcher: hconnection-0x236738a263a Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626) at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211) at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325) at
[jira] [Updated] (HBASE-5781) Zookeeper session got closed while trying to assign the region to RS using hbck -fix
[ https://issues.apache.org/jira/browse/HBASE-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5781: - Fix Version/s: 0.95.0 Zookeeper session got closed while trying to assign the region to RS using hbck -fix Key: HBASE-5781 URL: https://issues.apache.org/jira/browse/HBASE-5781 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.90.7, 0.92.1, 0.94.0, 0.95.2 Reporter: Kristam Subba Swathi Assignee: Jonathan Hsieh Priority: Critical Fix For: 0.92.2, 0.94.0, 0.95.0 Attachments: hbase-5781.patch After running the hbck in the cluster ,it is found that one region is not assigned So the hbck -fix is used to fix this But the assignment didnt happen since the zookeeper session is closed Please find the attached trace for more details - Trying to fix unassigned region... 12/04/03 11:02:57 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {NAME = 'ufdr,002300,179123498.00871fbd7583512e12c4eb38e900be8d.', STARTKEY = '002300', ENDKEY = '002311', ENCODED = 00871fbd7583512e12c4eb38e900be8d,} 12/04/03 11:02:58 INFO client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x236738a263a 12/04/03 11:02:58 INFO zookeeper.ZooKeeper: Session: 0x236738a263a closed ERROR: Region { meta = ufdr,010444,179123857.01594219211d0035b9586f98954462e1., hdfs = hdfs://10.18.40.25:9000/hbase/ufdr/01594219211d0035b9586f98954462e1, deployed = } not deployed on any region server. Trying to fix unassigned region... 12/04/03 11:02:58 INFO zookeeper.ClientCnxn: EventThread shut down 12/04/03 11:02:58 WARN zookeeper.ZKUtil: hconnection-0x236738a263a Unable to set watcher on znode (/hbase) org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626) at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211) at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325) at org.apache.hadoop.hbase.util.HBaseFsckRepair.forceOfflineInZK(HBaseFsckRepair.java:109) at org.apache.hadoop.hbase.util.HBaseFsckRepair.fixUnassigned(HBaseFsckRepair.java:92) at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1235) at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1351) at org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1114) at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:356) at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:375) at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2894) 12/04/03 11:02:58 ERROR zookeeper.ZooKeeperWatcher: hconnection-0x236738a263a Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626) at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211) at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325) at
[jira] [Updated] (HBASE-5781) Zookeeper session got closed while trying to assign the region to RS using hbck -fix
[ https://issues.apache.org/jira/browse/HBASE-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5781: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Anoop, Kristam, thanks for finding this and hunting down the root cause of the problem. Thanks for the quick review Lars. Committed to 0.90/0.92/0.94. Trunk is broken currently because of HBASE-5747, HBASE-5793. Zookeeper session got closed while trying to assign the region to RS using hbck -fix Key: HBASE-5781 URL: https://issues.apache.org/jira/browse/HBASE-5781 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.90.7, 0.92.1, 0.94.0, 0.96.0 Reporter: Kristam Subba Swathi Assignee: Jonathan Hsieh Priority: Critical Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0 Attachments: hbase-5781.patch After running the hbck in the cluster ,it is found that one region is not assigned So the hbck -fix is used to fix this But the assignment didnt happen since the zookeeper session is closed Please find the attached trace for more details - Trying to fix unassigned region... 12/04/03 11:02:57 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {NAME = 'ufdr,002300,179123498.00871fbd7583512e12c4eb38e900be8d.', STARTKEY = '002300', ENDKEY = '002311', ENCODED = 00871fbd7583512e12c4eb38e900be8d,} 12/04/03 11:02:58 INFO client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x236738a263a 12/04/03 11:02:58 INFO zookeeper.ZooKeeper: Session: 0x236738a263a closed ERROR: Region { meta = ufdr,010444,179123857.01594219211d0035b9586f98954462e1., hdfs = hdfs://10.18.40.25:9000/hbase/ufdr/01594219211d0035b9586f98954462e1, deployed = } not deployed on any region server. Trying to fix unassigned region... 12/04/03 11:02:58 INFO zookeeper.ClientCnxn: EventThread shut down 12/04/03 11:02:58 WARN zookeeper.ZKUtil: hconnection-0x236738a263a Unable to set watcher on znode (/hbase) org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626) at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211) at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325) at org.apache.hadoop.hbase.util.HBaseFsckRepair.forceOfflineInZK(HBaseFsckRepair.java:109) at org.apache.hadoop.hbase.util.HBaseFsckRepair.fixUnassigned(HBaseFsckRepair.java:92) at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1235) at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1351) at org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1114) at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:356) at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:375) at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2894) 12/04/03 11:02:58 ERROR zookeeper.ZooKeeperWatcher: hconnection-0x236738a263a Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695) at
[jira] [Updated] (HBASE-5781) Zookeeper session got closed while trying to assign the region to RS using hbck -fix
[ https://issues.apache.org/jira/browse/HBASE-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-5781: -- Affects Version/s: 0.94.0 Zookeeper session got closed while trying to assign the region to RS using hbck -fix Key: HBASE-5781 URL: https://issues.apache.org/jira/browse/HBASE-5781 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0 Reporter: Kristam Subba Swathi Assignee: Jonathan Hsieh After running the hbck in the cluster ,it is found that one region is not assigned So the hbck -fix is used to fix this But the assignment didnt happen since the zookeeper session is closed Please find the attached trace for more details - Trying to fix unassigned region... 12/04/03 11:02:57 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {NAME = 'ufdr,002300,179123498.00871fbd7583512e12c4eb38e900be8d.', STARTKEY = '002300', ENDKEY = '002311', ENCODED = 00871fbd7583512e12c4eb38e900be8d,} 12/04/03 11:02:58 INFO client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x236738a263a 12/04/03 11:02:58 INFO zookeeper.ZooKeeper: Session: 0x236738a263a closed ERROR: Region { meta = ufdr,010444,179123857.01594219211d0035b9586f98954462e1., hdfs = hdfs://10.18.40.25:9000/hbase/ufdr/01594219211d0035b9586f98954462e1, deployed = } not deployed on any region server. Trying to fix unassigned region... 12/04/03 11:02:58 INFO zookeeper.ClientCnxn: EventThread shut down 12/04/03 11:02:58 WARN zookeeper.ZKUtil: hconnection-0x236738a263a Unable to set watcher on znode (/hbase) org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626) at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211) at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325) at org.apache.hadoop.hbase.util.HBaseFsckRepair.forceOfflineInZK(HBaseFsckRepair.java:109) at org.apache.hadoop.hbase.util.HBaseFsckRepair.fixUnassigned(HBaseFsckRepair.java:92) at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1235) at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1351) at org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1114) at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:356) at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:375) at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2894) 12/04/03 11:02:58 ERROR zookeeper.ZooKeeperWatcher: hconnection-0x236738a263a Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626) at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211) at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325) at org.apache.hadoop.hbase.util.HBaseFsckRepair.forceOfflineInZK(HBaseFsckRepair.java:109) at org.apache.hadoop.hbase.util.HBaseFsckRepair.fixUnassigned(HBaseFsckRepair.java:92) at
[jira] [Updated] (HBASE-5781) Zookeeper session got closed while trying to assign the region to RS using hbck -fix
[ https://issues.apache.org/jira/browse/HBASE-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5781: -- Priority: Critical (was: Major) Affects Version/s: 0.96.0 Fix Version/s: 0.94.0 Zookeeper session got closed while trying to assign the region to RS using hbck -fix Key: HBASE-5781 URL: https://issues.apache.org/jira/browse/HBASE-5781 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Kristam Subba Swathi Assignee: Jonathan Hsieh Priority: Critical Fix For: 0.94.0 After running the hbck in the cluster ,it is found that one region is not assigned So the hbck -fix is used to fix this But the assignment didnt happen since the zookeeper session is closed Please find the attached trace for more details - Trying to fix unassigned region... 12/04/03 11:02:57 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {NAME = 'ufdr,002300,179123498.00871fbd7583512e12c4eb38e900be8d.', STARTKEY = '002300', ENDKEY = '002311', ENCODED = 00871fbd7583512e12c4eb38e900be8d,} 12/04/03 11:02:58 INFO client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x236738a263a 12/04/03 11:02:58 INFO zookeeper.ZooKeeper: Session: 0x236738a263a closed ERROR: Region { meta = ufdr,010444,179123857.01594219211d0035b9586f98954462e1., hdfs = hdfs://10.18.40.25:9000/hbase/ufdr/01594219211d0035b9586f98954462e1, deployed = } not deployed on any region server. Trying to fix unassigned region... 12/04/03 11:02:58 INFO zookeeper.ClientCnxn: EventThread shut down 12/04/03 11:02:58 WARN zookeeper.ZKUtil: hconnection-0x236738a263a Unable to set watcher on znode (/hbase) org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626) at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211) at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325) at org.apache.hadoop.hbase.util.HBaseFsckRepair.forceOfflineInZK(HBaseFsckRepair.java:109) at org.apache.hadoop.hbase.util.HBaseFsckRepair.fixUnassigned(HBaseFsckRepair.java:92) at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1235) at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1351) at org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1114) at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:356) at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:375) at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2894) 12/04/03 11:02:58 ERROR zookeeper.ZooKeeperWatcher: hconnection-0x236738a263a Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626) at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211) at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325) at
[jira] [Updated] (HBASE-5781) Zookeeper session got closed while trying to assign the region to RS using hbck -fix
[ https://issues.apache.org/jira/browse/HBASE-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5781: -- Affects Version/s: 0.90.7 0.92.1 Zookeeper session got closed while trying to assign the region to RS using hbck -fix Key: HBASE-5781 URL: https://issues.apache.org/jira/browse/HBASE-5781 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.90.7, 0.92.1, 0.94.0, 0.96.0 Reporter: Kristam Subba Swathi Assignee: Jonathan Hsieh Priority: Critical Fix For: 0.94.0 After running the hbck in the cluster ,it is found that one region is not assigned So the hbck -fix is used to fix this But the assignment didnt happen since the zookeeper session is closed Please find the attached trace for more details - Trying to fix unassigned region... 12/04/03 11:02:57 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {NAME = 'ufdr,002300,179123498.00871fbd7583512e12c4eb38e900be8d.', STARTKEY = '002300', ENDKEY = '002311', ENCODED = 00871fbd7583512e12c4eb38e900be8d,} 12/04/03 11:02:58 INFO client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x236738a263a 12/04/03 11:02:58 INFO zookeeper.ZooKeeper: Session: 0x236738a263a closed ERROR: Region { meta = ufdr,010444,179123857.01594219211d0035b9586f98954462e1., hdfs = hdfs://10.18.40.25:9000/hbase/ufdr/01594219211d0035b9586f98954462e1, deployed = } not deployed on any region server. Trying to fix unassigned region... 12/04/03 11:02:58 INFO zookeeper.ClientCnxn: EventThread shut down 12/04/03 11:02:58 WARN zookeeper.ZKUtil: hconnection-0x236738a263a Unable to set watcher on znode (/hbase) org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626) at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211) at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325) at org.apache.hadoop.hbase.util.HBaseFsckRepair.forceOfflineInZK(HBaseFsckRepair.java:109) at org.apache.hadoop.hbase.util.HBaseFsckRepair.fixUnassigned(HBaseFsckRepair.java:92) at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1235) at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1351) at org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1114) at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:356) at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:375) at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2894) 12/04/03 11:02:58 ERROR zookeeper.ZooKeeperWatcher: hconnection-0x236738a263a Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626) at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211) at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325) at
[jira] [Updated] (HBASE-5781) Zookeeper session got closed while trying to assign the region to RS using hbck -fix
[ https://issues.apache.org/jira/browse/HBASE-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5781: -- Attachment: hbase-5781.patch Implemented fix suggested in conversation. Applied to 0.92.x based hbase, and confirmed that hbck's assignment operations worked. * Fixed a borked test cluster * On ok cluster, use hbase shell to closed a region, ran updated hbck to verify detected, ran 'hbck -fix' to fix assignment and problem was repaired. Note for this to pass on trunk, HBASE-5993 is needed as well. Zookeeper session got closed while trying to assign the region to RS using hbck -fix Key: HBASE-5781 URL: https://issues.apache.org/jira/browse/HBASE-5781 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.90.7, 0.92.1, 0.94.0, 0.96.0 Reporter: Kristam Subba Swathi Assignee: Jonathan Hsieh Priority: Critical Fix For: 0.94.0 Attachments: hbase-5781.patch After running the hbck in the cluster ,it is found that one region is not assigned So the hbck -fix is used to fix this But the assignment didnt happen since the zookeeper session is closed Please find the attached trace for more details - Trying to fix unassigned region... 12/04/03 11:02:57 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {NAME = 'ufdr,002300,179123498.00871fbd7583512e12c4eb38e900be8d.', STARTKEY = '002300', ENDKEY = '002311', ENCODED = 00871fbd7583512e12c4eb38e900be8d,} 12/04/03 11:02:58 INFO client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x236738a263a 12/04/03 11:02:58 INFO zookeeper.ZooKeeper: Session: 0x236738a263a closed ERROR: Region { meta = ufdr,010444,179123857.01594219211d0035b9586f98954462e1., hdfs = hdfs://10.18.40.25:9000/hbase/ufdr/01594219211d0035b9586f98954462e1, deployed = } not deployed on any region server. Trying to fix unassigned region... 12/04/03 11:02:58 INFO zookeeper.ClientCnxn: EventThread shut down 12/04/03 11:02:58 WARN zookeeper.ZKUtil: hconnection-0x236738a263a Unable to set watcher on znode (/hbase) org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626) at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211) at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325) at org.apache.hadoop.hbase.util.HBaseFsckRepair.forceOfflineInZK(HBaseFsckRepair.java:109) at org.apache.hadoop.hbase.util.HBaseFsckRepair.fixUnassigned(HBaseFsckRepair.java:92) at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1235) at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1351) at org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1114) at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:356) at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:375) at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2894) 12/04/03 11:02:58 ERROR zookeeper.ZooKeeperWatcher: hconnection-0x236738a263a Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208) at
[jira] [Updated] (HBASE-5781) Zookeeper session got closed while trying to assign the region to RS using hbck -fix
[ https://issues.apache.org/jira/browse/HBASE-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5781: -- Fix Version/s: 0.96.0 0.92.2 0.90.7 Status: Patch Available (was: Open) Zookeeper session got closed while trying to assign the region to RS using hbck -fix Key: HBASE-5781 URL: https://issues.apache.org/jira/browse/HBASE-5781 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.92.1, 0.90.7, 0.94.0, 0.96.0 Reporter: Kristam Subba Swathi Assignee: Jonathan Hsieh Priority: Critical Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0 Attachments: hbase-5781.patch After running the hbck in the cluster ,it is found that one region is not assigned So the hbck -fix is used to fix this But the assignment didnt happen since the zookeeper session is closed Please find the attached trace for more details - Trying to fix unassigned region... 12/04/03 11:02:57 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {NAME = 'ufdr,002300,179123498.00871fbd7583512e12c4eb38e900be8d.', STARTKEY = '002300', ENDKEY = '002311', ENCODED = 00871fbd7583512e12c4eb38e900be8d,} 12/04/03 11:02:58 INFO client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x236738a263a 12/04/03 11:02:58 INFO zookeeper.ZooKeeper: Session: 0x236738a263a closed ERROR: Region { meta = ufdr,010444,179123857.01594219211d0035b9586f98954462e1., hdfs = hdfs://10.18.40.25:9000/hbase/ufdr/01594219211d0035b9586f98954462e1, deployed = } not deployed on any region server. Trying to fix unassigned region... 12/04/03 11:02:58 INFO zookeeper.ClientCnxn: EventThread shut down 12/04/03 11:02:58 WARN zookeeper.ZKUtil: hconnection-0x236738a263a Unable to set watcher on znode (/hbase) org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626) at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211) at org.apache.hadoop.hbase.client.HBaseAdmin.assign(HBaseAdmin.java:1325) at org.apache.hadoop.hbase.util.HBaseFsckRepair.forceOfflineInZK(HBaseFsckRepair.java:109) at org.apache.hadoop.hbase.util.HBaseFsckRepair.fixUnassigned(HBaseFsckRepair.java:92) at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1235) at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1351) at org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1114) at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:356) at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:375) at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:2894) 12/04/03 11:02:58 ERROR zookeeper.ZooKeeperWatcher: hconnection-0x236738a263a Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:263) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(HConnectionManager.java:695) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:626) at org.apache.hadoop.hbase.client.HBaseAdmin.getMaster(HBaseAdmin.java:211) at