[jira] [Updated] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-20334: Resolution: Fixed Release Note: HBase now includes a helper script that can be used to run a basic functionality test for a given HBase installation at in `dev_support`. The test can optionally be given an HBase client artifact to rely on and can optionally be given specific Hadoop client artifacts to use. For usage information see `./dev-support/hbase_nightly_pseudo-distributed-test.sh --help`. The project nightly tests now make use of this test to check running on top of Hadoop 2, Hadoop 3, and Hadoop 3 with shaded client artifacts. Status: Resolved (was: Patch Available) > add a test that expressly uses both our shaded client and the one from hadoop > 3 > --- > > Key: HBASE-20334 > URL: https://issues.apache.org/jira/browse/HBASE-20334 > Project: HBase > Issue Type: Sub-task > Components: hadoop3, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch > > > Since we're making a shaded client that bleed out of our namespace and into > Hadoop's, we should ensure that we can show our clients coexisting. Even if > it's just an IT that successfully talks to both us and HDFS via our > respective shaded clients, that'd be a big help in keeping us proactive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-20334: Fix Version/s: 2.1.0 3.0.0 > add a test that expressly uses both our shaded client and the one from hadoop > 3 > --- > > Key: HBASE-20334 > URL: https://issues.apache.org/jira/browse/HBASE-20334 > Project: HBase > Issue Type: Sub-task > Components: hadoop3, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch > > > Since we're making a shaded client that bleed out of our namespace and into > Hadoop's, we should ensure that we can show our clients coexisting. Even if > it's just an IT that successfully talks to both us and HDFS via our > respective shaded clients, that'd be a big help in keeping us proactive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-20334: Status: Patch Available (was: In Progress) v1 - rebased master + patches for HBASE-20332, HBASE-20333, HBASE-19735, and HBASE-20615 - adds a support script for automating caching apache project artifacts - updates Jenkinsfile to use said caching script ot get a copy of hadoop 2 and hadoop 3 - updates our "source tarball" test to be general "packaging and integration with Hadoop" - adds use of test script with hadoop 2, hadoop 3, and hadoop 3 w/hadoop's client artifacts - updated handling of args to the test script so that we work with relative paths - updated for feedback from v0: does file checks for passed jars. > add a test that expressly uses both our shaded client and the one from hadoop > 3 > --- > > Key: HBASE-20334 > URL: https://issues.apache.org/jira/browse/HBASE-20334 > Project: HBase > Issue Type: Sub-task > Components: hadoop3, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch > > > Since we're making a shaded client that bleed out of our namespace and into > Hadoop's, we should ensure that we can show our clients coexisting. Even if > it's just an IT that successfully talks to both us and HDFS via our > respective shaded clients, that'd be a big help in keeping us proactive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-20334: Attachment: HBASE-20334.1.patch > add a test that expressly uses both our shaded client and the one from hadoop > 3 > --- > > Key: HBASE-20334 > URL: https://issues.apache.org/jira/browse/HBASE-20334 > Project: HBase > Issue Type: Sub-task > Components: hadoop3, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch > > > Since we're making a shaded client that bleed out of our namespace and into > Hadoop's, we should ensure that we can show our clients coexisting. Even if > it's just an IT that successfully talks to both us and HDFS via our > respective shaded clients, that'd be a big help in keeping us proactive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-20334: Status: In Progress (was: Patch Available) reworking this and HBASE-20615 to build on HBASE-19735 > add a test that expressly uses both our shaded client and the one from hadoop > 3 > --- > > Key: HBASE-20334 > URL: https://issues.apache.org/jira/browse/HBASE-20334 > Project: HBase > Issue Type: Sub-task > Components: hadoop3, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Attachments: HBASE-20334.0.patch > > > Since we're making a shaded client that bleed out of our namespace and into > Hadoop's, we should ensure that we can show our clients coexisting. Even if > it's just an IT that successfully talks to both us and HDFS via our > respective shaded clients, that'd be a big help in keeping us proactive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20334:
Status: Patch Available (was: In Progress)
Here's a WIP. It relies on HBASE-20332, HBASE-20333, and HBASE-20615 so that
patch has 4 commits.
-v0
- adds hbase_nightly_pseudo-distributed-test.sh
Test takes as required params:
- path to an hbase convenience binary tarball
- path to a {{hadoop}} executable
- path to the hadoop hadoop-yarn-server-tests-tests and
hadoop-mapreduce-client-jobclient-tests jars (needed to run the hadoop mini
cluster)
Test takes optionally:
- a working directory, defaulting to making something in /tmp (similar to our
other dev-support scripts)
- a location for zk data (defaulting to making something in the working dir)
- wether or not to remove data from previous tests
- a classpath for hadoop clients (defaulting to using the {{hadoop classpath}}
command)
- wether to be a single process or pseudodistributed
Checked so far, each as single process, each takes ~5 minutes on my local
laptop:
- Against Hadoop 2.7.3 installed by homebrew
{code}
time ./dev-support/hbase_nightly_pseudo-distributed-test.sh --single-process
hbase-assembly/target/hbase-3.0.0-SNAPSHOT/
/usr/local/Cellar/hadoop/2.7.3/bin/hadoop
/usr/local/Cellar/hadoop/2.7.3/libexec/share/hadoop/yarn/test/hadoop-yarn-server-tests-2.7.3-tests.jar
/usr/local/Cellar/hadoop/2.7.3/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
{code}
- Against Hadoop 3.0.0 installed by homebrew
{code}
time ./dev-support/hbase_nightly_pseudo-distributed-test.sh --single-process
hbase-assembly/target/hbase-3.0.0-SNAPSHOT/
/usr/local/Cellar/hadoop/3.0.0/bin/hadoop
/usr/local/Cellar/hadoop/3.0.0/libexec/share/hadoop/yarn/test/hadoop-yarn-server-tests-3.0.0-tests.jar
/usr/local/Cellar/hadoop/3.0.0/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-tests.jar
{code}
- Against Hadoop 3.0.0 w/shaded hadoop artifacts, installed by homebrew
{code}
time ./dev-support/hbase_nightly_pseudo-distributed-test.sh
--hadoop-client-classpath
/usr/local/Cellar/hadoop/3.0.0/libexec/share/hadoop/client/hadoop-client-api-3.0.0.jar:/usr/local/Cellar/hadoop/3.0.0/libexec/share/hadoop/client/hadoop-client-runtime-3.0.0.jar
--single-process hbase-assembly/target/hbase-3.0.0-SNAPSHOT/
/usr/local/Cellar/hadoop/3.0.0/bin/hadoop
/usr/local/Cellar/hadoop/3.0.0/libexec/share/hadoop/yarn/test/hadoop-yarn-server-tests-3.0.0-tests.jar
/usr/local/Cellar/hadoop/3.0.0/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-tests.jar
{code}
TODO:
- Jenkinsfile addition, including getting Hadoop artifacts
- multiple process (might be blocked by HBASE-6555)
> add a test that expressly uses both our shaded client and the one from hadoop
> 3
> ---
>
> Key: HBASE-20334
> URL: https://issues.apache.org/jira/browse/HBASE-20334
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20334.0.patch
>
>
> Since we're making a shaded client that bleed out of our namespace and into
> Hadoop's, we should ensure that we can show our clients coexisting. Even if
> it's just an IT that successfully talks to both us and HDFS via our
> respective shaded clients, that'd be a big help in keeping us proactive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-20334: Attachment: HBASE-20334.0.patch > add a test that expressly uses both our shaded client and the one from hadoop > 3 > --- > > Key: HBASE-20334 > URL: https://issues.apache.org/jira/browse/HBASE-20334 > Project: HBase > Issue Type: Sub-task > Components: hadoop3, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Attachments: HBASE-20334.0.patch > > > Since we're making a shaded client that bleed out of our namespace and into > Hadoop's, we should ensure that we can show our clients coexisting. Even if > it's just an IT that successfully talks to both us and HDFS via our > respective shaded clients, that'd be a big help in keeping us proactive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
