[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20334:
--------------------------------
Status: Patch Available (was: In Progress)
Here's a WIP. It relies on HBASE-20332, HBASE-20333, and HBASE-20615 so that
patch has 4 commits.
-v0
- adds hbase_nightly_pseudo-distributed-test.sh
Test takes as required params:
- path to an hbase convenience binary tarball
- path to a {{hadoop}} executable
- path to the hadoop hadoop-yarn-server-tests-tests and
hadoop-mapreduce-client-jobclient-tests jars (needed to run the hadoop mini
cluster)
Test takes optionally:
- a working directory, defaulting to making something in /tmp (similar to our
other dev-support scripts)
- a location for zk data (defaulting to making something in the working dir)
- wether or not to remove data from previous tests
- a classpath for hadoop clients (defaulting to using the {{hadoop classpath}}
command)
- wether to be a single process or pseudodistributed
Checked so far, each as single process, each takes ~5 minutes on my local
laptop:
- Against Hadoop 2.7.3 installed by homebrew
{code}
time ./dev-support/hbase_nightly_pseudo-distributed-test.sh --single-process
hbase-assembly/target/hbase-3.0.0-SNAPSHOT/
/usr/local/Cellar/hadoop/2.7.3/bin/hadoop
/usr/local/Cellar/hadoop/2.7.3/libexec/share/hadoop/yarn/test/hadoop-yarn-server-tests-2.7.3-tests.jar
/usr/local/Cellar/hadoop/2.7.3/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
{code}
- Against Hadoop 3.0.0 installed by homebrew
{code}
time ./dev-support/hbase_nightly_pseudo-distributed-test.sh --single-process
hbase-assembly/target/hbase-3.0.0-SNAPSHOT/
/usr/local/Cellar/hadoop/3.0.0/bin/hadoop
/usr/local/Cellar/hadoop/3.0.0/libexec/share/hadoop/yarn/test/hadoop-yarn-server-tests-3.0.0-tests.jar
/usr/local/Cellar/hadoop/3.0.0/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-tests.jar
{code}
- Against Hadoop 3.0.0 w/shaded hadoop artifacts, installed by homebrew
{code}
time ./dev-support/hbase_nightly_pseudo-distributed-test.sh
--hadoop-client-classpath
/usr/local/Cellar/hadoop/3.0.0/libexec/share/hadoop/client/hadoop-client-api-3.0.0.jar:/usr/local/Cellar/hadoop/3.0.0/libexec/share/hadoop/client/hadoop-client-runtime-3.0.0.jar
--single-process hbase-assembly/target/hbase-3.0.0-SNAPSHOT/
/usr/local/Cellar/hadoop/3.0.0/bin/hadoop
/usr/local/Cellar/hadoop/3.0.0/libexec/share/hadoop/yarn/test/hadoop-yarn-server-tests-3.0.0-tests.jar
/usr/local/Cellar/hadoop/3.0.0/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-tests.jar
{code}
TODO:
- Jenkinsfile addition, including getting Hadoop artifacts
- multiple process (might be blocked by HBASE-6555)
> add a test that expressly uses both our shaded client and the one from hadoop
> 3
> -------------------------------------------------------------------------------
>
> Key: HBASE-20334
> URL: https://issues.apache.org/jira/browse/HBASE-20334
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3, shading
> Affects Versions: 2.0.0
> Reporter: Sean Busbey
> Assignee: Sean Busbey
> Priority: Major
> Attachments: HBASE-20334.0.patch
>
>
> Since we're making a shaded client that bleed out of our namespace and into
> Hadoop's, we should ensure that we can show our clients coexisting. Even if
> it's just an IT that successfully talks to both us and HDFS via our
> respective shaded clients, that'd be a big help in keeping us proactive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
