[
https://issues.apache.org/jira/browse/AMBARI-11743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Fernandez updated AMBARI-11743:
-----------------------------------------
Description:
1. Install cluster with Ambari 2.1 and HDP 2.3
2. Add services HDFS, YARN, MR, ZK, and HBaste
3. Perform several Stop All and Start All on HDFS service
4. Periodically, HBase Master will crash
This was a non-HA cluster.
{code}
2015-06-02 09:34:24,865 WARN [ip-172-31-33-225:16000.activeMasterManager]
hdfs.DFSClient: Could not obtain block:
BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005
file=/apps/hbase/data/hbase.id No live nodes contain current block Block
locations: Dead nodes: . Throwing a BlockMissingException
2015-06-02 09:34:24,866 WARN [ip-172-31-33-225:16000.activeMasterManager]
hdfs.DFSClient: DFS Read
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block:
BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005
file=/apps/hbase/data/hbase.id
at
org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945)
at
org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604)
at
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:816)
at
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:474)
at
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
at
org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
at java.lang.Thread.run(Thread.java:745)
2015-06-02 09:34:24,870 FATAL [ip-172-31-33-225:16000.activeMasterManager]
master.HMaster: Failed to become active master
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block:
BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005
file=/apps/hbase/data/hbase.id
at
org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945)
at
org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604)
at
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:816)
at
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:474)
at
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
at
org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
at java.lang.Thread.run(Thread.java:745)
{code}
was:
1. Install cluster with Ambari 2.1 and HDP 2.3
2. Add services HDFS, YARN, MR, ZK, and HBaste
3. Perform several Stop All and Start All on HDFS service
4. Periodically, HBase Master will crash
This was a non-HA cluster.
> NameNode is forced to leave safemode, which causes HBMaster master to crash
> if done too quickly
> -----------------------------------------------------------------------------------------------
>
> Key: AMBARI-11743
> URL: https://issues.apache.org/jira/browse/AMBARI-11743
> Project: Ambari
> Issue Type: Bug
> Reporter: Alejandro Fernandez
> Assignee: Alejandro Fernandez
>
> 1. Install cluster with Ambari 2.1 and HDP 2.3
> 2. Add services HDFS, YARN, MR, ZK, and HBaste
> 3. Perform several Stop All and Start All on HDFS service
> 4. Periodically, HBase Master will crash
> This was a non-HA cluster.
> {code}
> 2015-06-02 09:34:24,865 WARN [ip-172-31-33-225:16000.activeMasterManager]
> hdfs.DFSClient: Could not obtain block:
> BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005
> file=/apps/hbase/data/hbase.id No live nodes contain current block Block
> locations: Dead nodes: . Throwing a BlockMissingException
> 2015-06-02 09:34:24,866 WARN [ip-172-31-33-225:16000.activeMasterManager]
> hdfs.DFSClient: DFS Read
> org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block:
> BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005
> file=/apps/hbase/data/hbase.id
> at
> org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945)
> at
> org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
> at java.io.DataInputStream.readFully(DataInputStream.java:195)
> at java.io.DataInputStream.readFully(DataInputStream.java:169)
> at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:816)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:474)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
> at java.lang.Thread.run(Thread.java:745)
> 2015-06-02 09:34:24,870 FATAL [ip-172-31-33-225:16000.activeMasterManager]
> master.HMaster: Failed to become active master
> org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block:
> BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005
> file=/apps/hbase/data/hbase.id
> at
> org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945)
> at
> org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
> at java.io.DataInputStream.readFully(DataInputStream.java:195)
> at java.io.DataInputStream.readFully(DataInputStream.java:169)
> at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:816)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:474)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
