Loknath Priyatham Teja Singamsetty created HBASE-16375:
-----------------------------------------------------------
Summary: Mapreduce mini cluster using HBaseTestingUtility not
setting correct resourcemanager and jobhistory webapp address of
MapReduceTestingShim
Key: HBASE-16375
URL: https://issues.apache.org/jira/browse/HBASE-16375
Project: HBase
Issue Type: Bug
Reporter: Loknath Priyatham Teja Singamsetty
Assignee: Loknath Priyatham Teja Singamsetty
Starting mapreduce mini cluster using HBaseTestingUtility is not setting
"yarn.resourcemanager.webapp.address" and "mapreduce.jobhistory.webapp.address"
which are required for getting the submitted yarn apps using mapreduce webapp.
These properties are not being copied from jobConf of MapReduceTestingShim
resulting in default values.
{quote}
HBaseTestingUtility.java
// Allow the user to override FS URI for this map-reduce cluster to use.
mrCluster = new MiniMRCluster(servers,
FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
null, null, new JobConf(this.conf));
JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
if (jobConf == null) {
jobConf = mrCluster.createJobConf();
}
jobConf.set("mapreduce.cluster.local.dir",
conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites this
while it should not
LOG.info("Mini mapreduce cluster started");
// In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
and updates settings.
// Our HBase MR jobs need several of these settings in order to properly
run. So we copy the
// necessary config properties here. YARN-129 required adding a few
properties.
conf.set("mapreduce.jobtracker.address",
jobConf.get("mapreduce.jobtracker.address"));
// this for mrv2 support; mr1 ignores this
conf.set("mapreduce.framework.name", "yarn");
conf.setBoolean("yarn.is.minicluster", true);
String rmAddress = jobConf.get("yarn.resourcemanager.address");
if (rmAddress != null) {
conf.set("yarn.resourcemanager.address", rmAddress);
}
String historyAddress = jobConf.get("mapreduce.jobhistory.address");
if (historyAddress != null) {
conf.set("mapreduce.jobhistory.address", historyAddress);
}
String schedulerAddress =
jobConf.get("yarn.resourcemanager.scheduler.address");
if (schedulerAddress != null) {
conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
}
{quote}
As a immediate fix for phoenix e2e test to succeed, need the below lines to be
added as well
{quote}
String rmWebappAddress = jobConf.get("yarn.resourcemanager.webapp.address");
if (rmWebappAddress != null) {
conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
}
String historyWebappAddress =
jobConf.get("mapreduce.jobhistory.webapp.address");
if (historyWebappAddress != null) {
conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
}
{quote}
Eventually, we should also see if we can copy over all the jobConf properties
to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
