[jira] [Updated] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-16375:
-
Fix Version/s: (was: 1.1.6)
1.1.7
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 0.98.22, 1.1.7, 1.2.4
>
> Attachments: HBASE-16375.master.001.patch, HBASE-16375.patch,
> HBASE-16375_0.98_and_above.patch,
> HBASE-16375_0.98_and_above_with_tests.patch,
> HBASE-16375_0.98_and_above_with_tests_format.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-16375:
---
Attachment: HBASE-16375.patch
Attaching what I committed as 'HBASE-16375.patch'
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.1.6, 0.98.22, 1.2.4
>
> Attachments: HBASE-16375.master.001.patch, HBASE-16375.patch,
> HBASE-16375_0.98_and_above.patch,
> HBASE-16375_0.98_and_above_with_tests.patch,
> HBASE-16375_0.98_and_above_with_tests_format.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-16375:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.1.6, 0.98.22, 1.2.4
>
> Attachments: HBASE-16375.master.001.patch,
> HBASE-16375_0.98_and_above.patch,
> HBASE-16375_0.98_and_above_with_tests.patch,
> HBASE-16375_0.98_and_above_with_tests_format.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Loknath Priyatham Teja Singamsetty updated HBASE-16375:
Attachment: HBASE-16375.master.001.patch
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.1.6, 0.98.22, 1.2.4
>
> Attachments: HBASE-16375.master.001.patch,
> HBASE-16375_0.98_and_above.patch,
> HBASE-16375_0.98_and_above_with_tests.patch,
> HBASE-16375_0.98_and_above_with_tests_format.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Loknath Priyatham Teja Singamsetty updated HBASE-16375:
Attachment: HBASE-16375_0.98_and_above_with_tests_format.patch
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.1.6, 0.98.22, 1.2.4
>
> Attachments: HBASE-16375_0.98_and_above.patch,
> HBASE-16375_0.98_and_above_with_tests.patch,
> HBASE-16375_0.98_and_above_with_tests_format.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Loknath Priyatham Teja Singamsetty updated HBASE-16375:
Attachment: HBASE-16375_0.98_and_above_with_tests.patch
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.1.6, 0.98.22, 1.2.4
>
> Attachments: HBASE-16375_0.98_and_above.patch,
> HBASE-16375_0.98_and_above_with_tests.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-16375:
--
Fix Version/s: (was: 1.2.3)
(was: 1.3.1)
(was: 1.2.2)
(was: 1.4.0)
(was: 1.2.0)
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.1.6, 0.98.22, 1.2.4
>
> Attachments: HBASE-16375_0.98_and_above.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Loknath Priyatham Teja Singamsetty updated HBASE-16375:
Attachment: (was: HBASE-16375_1.2.0_and_above.patch)
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.4.0, 1.2.2, 1.1.6, 1.3.1, 1.2.3,
> 0.98.22, 1.2.4
>
> Attachments: HBASE-16375_0.98_and_above.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Loknath Priyatham Teja Singamsetty updated HBASE-16375:
Attachment: HBASE-16375_0.98_and_above.patch
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.4.0, 1.2.2, 1.1.6, 1.3.1, 1.2.3,
> 0.98.22, 1.2.4
>
> Attachments: HBASE-16375_0.98_and_above.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Loknath Priyatham Teja Singamsetty updated HBASE-16375:
Fix Version/s: 0.98.22
1.1.6
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.4.0, 1.2.2, 1.1.6, 1.3.1, 1.2.3,
> 0.98.22, 1.2.4
>
> Attachments: HBASE-16375_1.2.0_and_above.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Loknath Priyatham Teja Singamsetty updated HBASE-16375:
Attachment: HBASE-16375_1.2.0_and_above.patch
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.4.0, 1.2.2, 1.3.1, 1.2.3, 1.2.4
>
> Attachments: HBASE-16375_1.2.0_and_above.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Loknath Priyatham Teja Singamsetty updated HBASE-16375:
Fix Version/s: 1.3.1
1.4.0
1.3.0
2.0.0
Status: Patch Available (was: Open)
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.3.1, 1.2.3, 1.2.4, 1.2.2, 1.2.0
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Loknath Priyatham Teja Singamsetty updated HBASE-16375:
Fix Version/s: 1.2.0
1.2.2
1.2.4
1.2.3
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 1.2.0, 1.2.2, 1.2.3, 1.2.4
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Loknath Priyatham Teja Singamsetty updated HBASE-16375:
Affects Version/s: 1.2.0
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Loknath Priyatham Teja Singamsetty updated HBASE-16375:
Priority: Minor (was: Major)
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
