[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13624084#comment-13624084
]
Nick Dimiduk commented on HBASE-8258:
-------------------------------------
bq. My fear is that there are fixes we need in upstream so lets figure them out
sooner rather than later
Good point.
bq. copyConfigurationValues seems overly broad and doesn't expose the root
cause.
Cluster management in this test is pretty difficult to follow. I think the
underlying cause is that the job configuration is built against the wrong
cluster config. Let's see if I understand this correctly.
Back in {{beforeClass}} we have:
{code}
@BeforeClass
public static void beforeClass() throws Exception {
cluster = UTIL.startMiniCluster();
UTIL.startMiniMapReduceCluster();
}
{code}
I guess these are two different clusters, one running HBase, managed via
{{cluster}}, and the other running MR, managed by {{UTIL}}.
{{testSimpleCase}} starts by populating a table on {{cluster}} (HBase cluster),
it then creates the Export job using the configuration of {{cluster}} (HBase
cluster), even though that's the cluster lacking MR. Thus, it's necessary to
copy the MR details from {{UTIL}} (MR cluster) into the Export job's config.
Since the test is, at it's essence, running a MR job, wouldn't the test be made
more stable (and realistic to user jobs) if instead the HBase connection
details were copied into a MR job config build from {{UTIL}} (MR cluster)?
Further, that copy can be done in one place, way back in {{beforeClass}}. That
way, the code repeated for each test is limited.
{code}
// in beforeClass:
this.baseConf = new Configuration(UTIL.getConfiguration());
... // copy hbase configs into baseConf
// in each test
GenericOptionsParser opts = new GenericOptionsParser(new
Configuration(baseConf), args);
Configuration exportConf = opts.getConfiguration();
args = opts.getRemainingArgs();
{code}
This can be further simplified if the jobs under test implement the {{Tool}}
interface (like I did for ImportTsv in HBASE-8011) and deprecating the
{{createSubmittableJob}} nonsense. Then it becomes something closer to:
{code}
ToolRunner.run(new Configuration(baseConf), new Export(), args);
{code}
> Make mapreduce tests pass on hadoop2
> ------------------------------------
>
> Key: HBASE-8258
> URL: https://issues.apache.org/jira/browse/HBASE-8258
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
> Reporter: stack
> Priority: Blocker
> Fix For: 0.95.1
>
> Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt,
> 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch
>
>
> HBASE-7904 was a first attempt at making this work but it got lost in the
> weeds.
> This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
> breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
