[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16517690#comment-16517690
]
Hudson commented on HBASE-20334:
Results for branch master
[build #370 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/370/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/370//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/370//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/370//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> add a test that expressly uses both our shaded client and the one from hadoop
> 3
> ---
>
> Key: HBASE-20334
> URL: https://issues.apache.org/jira/browse/HBASE-20334
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch
>
>
> Since we're making a shaded client that bleed out of our namespace and into
> Hadoop's, we should ensure that we can show our clients coexisting. Even if
> it's just an IT that successfully talks to both us and HDFS via our
> respective shaded clients, that'd be a big help in keeping us proactive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516563#comment-16516563
]
Hudson commented on HBASE-20334:
Results for branch branch-2
[build #878 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/878/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/878//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/878//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/878//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> add a test that expressly uses both our shaded client and the one from hadoop
> 3
> ---
>
> Key: HBASE-20334
> URL: https://issues.apache.org/jira/browse/HBASE-20334
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch
>
>
> Since we're making a shaded client that bleed out of our namespace and into
> Hadoop's, we should ensure that we can show our clients coexisting. Even if
> it's just an IT that successfully talks to both us and HDFS via our
> respective shaded clients, that'd be a big help in keeping us proactive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16512749#comment-16512749
]
Josh Elser commented on HBASE-20334:
{quote}I got blocked by HBASE-6555. I think fixing that would be an incremental
improvement but not something worth blocking on.
{quote}
+1
> add a test that expressly uses both our shaded client and the one from hadoop
> 3
> ---
>
> Key: HBASE-20334
> URL: https://issues.apache.org/jira/browse/HBASE-20334
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch
>
>
> Since we're making a shaded client that bleed out of our namespace and into
> Hadoop's, we should ensure that we can show our clients coexisting. Even if
> it's just an IT that successfully talks to both us and HDFS via our
> respective shaded clients, that'd be a big help in keeping us proactive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510174#comment-16510174
]
Sean Busbey commented on HBASE-20334:
-
oh! almost forgot. the tests use {{--single-process}} because that's the one I
could use locally. My machine doesn't allow incoming SSH, so I got blocked by
HBASE-6555. I think fixing that would be an incremental improvement but not
something worth blocking on.
> add a test that expressly uses both our shaded client and the one from hadoop
> 3
> ---
>
> Key: HBASE-20334
> URL: https://issues.apache.org/jira/browse/HBASE-20334
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch
>
>
> Since we're making a shaded client that bleed out of our namespace and into
> Hadoop's, we should ensure that we can show our clients coexisting. Even if
> it's just an IT that successfully talks to both us and HDFS via our
> respective shaded clients, that'd be a big help in keeping us proactive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510164#comment-16510164 ] Sean Busbey commented on HBASE-20334: - looks like on ASF jenkins it adds ~4 minutes per configuration we test. so in the current patch that's +12 minutes. On top of the existing stuff in this test stage we're at a total of ~30minutes. well below the 4-5 hours the parallel yetus batteries take. > add a test that expressly uses both our shaded client and the one from hadoop > 3 > --- > > Key: HBASE-20334 > URL: https://issues.apache.org/jira/browse/HBASE-20334 > Project: HBase > Issue Type: Sub-task > Components: hadoop3, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch > > > Since we're making a shaded client that bleed out of our namespace and into > Hadoop's, we should ensure that we can show our clients coexisting. Even if > it's just an IT that successfully talks to both us and HDFS via our > respective shaded clients, that'd be a big help in keeping us proactive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510163#comment-16510163 ] Sean Busbey commented on HBASE-20334: - I forgot to click "keep this build" on the example above. so here's another: https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/45/ https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/45/artifact/output-integration/ It's already finished the test added here; it'll probably be 3 or 4 more hours for it to finish the other unrelated tests. > add a test that expressly uses both our shaded client and the one from hadoop > 3 > --- > > Key: HBASE-20334 > URL: https://issues.apache.org/jira/browse/HBASE-20334 > Project: HBase > Issue Type: Sub-task > Components: hadoop3, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch > > > Since we're making a shaded client that bleed out of our namespace and into > Hadoop's, we should ensure that we can show our clients coexisting. Even if > it's just an IT that successfully talks to both us and HDFS via our > respective shaded clients, that'd be a big help in keeping us proactive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16509068#comment-16509068
]
Sean Busbey commented on HBASE-20334:
-
the unit test failure was:
{code}
[ERROR]
org.apache.hadoop.hbase.replication.TestRemoveFromSerialReplicationPeer.org.apache.hadoop.hbase.replication.TestRemoveFromSerialReplicationPeer
[ERROR] Run 1:
TestRemoveFromSerialReplicationPeer.testRemoveSerialFlag:103->SerialReplicationTestBase.addPeer:219
» TestTimedOut
[ERROR] Run 2:
TestRemoveFromSerialReplicationPeer.org.apache.hadoop.hbase.replication.TestRemoveFromSerialReplicationPeer
»
[ERROR] Run 3:
TestRemoveFromSerialReplicationPeer>SerialReplicationTestBase.setUpBeforeClass:123
» IllegalState
[ERROR] Run 4:
TestRemoveFromSerialReplicationPeer>SerialReplicationTestBase.tearDownAfterClass:133
» IndexOutOfBounds
[ERROR] Run 5:
TestRemoveFromSerialReplicationPeer>SerialReplicationTestBase.setUpBeforeClass:123
» IllegalState
[INFO]
[ERROR]
TestRemoveFromSerialReplicationPeer.testRemoveSerialFlag:103->SerialReplicationTestBase.addPeer:219
» InterruptedIO
[ERROR]
TestRemoveFromSerialReplicationPeer>SerialReplicationTestBase.tearDown:140 »
TimeoutIO
[INFO]
[ERROR] Tests run: 3496, Failures: 0, Errors: 3, Skipped: 45
{code}
This change doesn't touch anything outside of {{dev-support}} so it couldn't
have caused this failure.
> add a test that expressly uses both our shaded client and the one from hadoop
> 3
> ---
>
> Key: HBASE-20334
> URL: https://issues.apache.org/jira/browse/HBASE-20334
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch
>
>
> Since we're making a shaded client that bleed out of our namespace and into
> Hadoop's, we should ensure that we can show our clients coexisting. Even if
> it's just an IT that successfully talks to both us and HDFS via our
> respective shaded clients, that'd be a big help in keeping us proactive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16509065#comment-16509065
]
Sean Busbey commented on HBASE-20334:
-
the shellcheck warning is this:
{code}
dev-support/hbase_nightly_pseudo-distributed-test.sh:202:18: warning: hbase is
referenced but not assigned. [SC2154]
{code}
It's a false positive hitting something in the HEREDOC for the test's
log4j.properties file.
> add a test that expressly uses both our shaded client and the one from hadoop
> 3
> ---
>
> Key: HBASE-20334
> URL: https://issues.apache.org/jira/browse/HBASE-20334
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch
>
>
> Since we're making a shaded client that bleed out of our namespace and into
> Hadoop's, we should ensure that we can show our clients coexisting. Even if
> it's just an IT that successfully talks to both us and HDFS via our
> respective shaded clients, that'd be a big help in keeping us proactive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16509064#comment-16509064
]
Sean Busbey commented on HBASE-20334:
-
{quote}
-1 whitespace 0m 0s The patch has 29 line(s) that end in
whitespace. Use git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply
-1 whitespace 0m 1s The patch 48 line(s) with tabs.
{quote}
The tabas are all in the TSV file used in the test. all but one of the
whitespace instances is also. the last one is an error in the commit for
HBASE-20333 that I'll fix either on commit or if that jira needs another patch.
> add a test that expressly uses both our shaded client and the one from hadoop
> 3
> ---
>
> Key: HBASE-20334
> URL: https://issues.apache.org/jira/browse/HBASE-20334
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch
>
>
> Since we're making a shaded client that bleed out of our namespace and into
> Hadoop's, we should ensure that we can show our clients coexisting. Even if
> it's just an IT that successfully talks to both us and HDFS via our
> respective shaded clients, that'd be a big help in keeping us proactive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16509063#comment-16509063 ] Sean Busbey commented on HBASE-20334: - Here's v1 running in a nightly branch: https://builds.apache.org/blue/organizations/jenkins/HBase%20Nightly/detail/HBASE-20331/43/pipeline/100 and here's the output logs of each of the three configurations that are tested: https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/43/artifact/output-integration/ > add a test that expressly uses both our shaded client and the one from hadoop > 3 > --- > > Key: HBASE-20334 > URL: https://issues.apache.org/jira/browse/HBASE-20334 > Project: HBase > Issue Type: Sub-task > Components: hadoop3, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch > > > Since we're making a shaded client that bleed out of our namespace and into > Hadoop's, we should ensure that we can show our clients coexisting. Even if > it's just an IT that successfully talks to both us and HDFS via our > respective shaded clients, that'd be a big help in keeping us proactive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16509049#comment-16509049
]
Hadoop QA commented on HBASE-20334:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
5s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
11s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
45s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
3s{color} | {color:red} The patch generated 1 new + 33 unchanged - 11 fixed =
34 total (was 44) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 29 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch 48 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
43s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
27s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 53s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}131m 4s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 8m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}198m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.replication.TestRemoveFromSerialReplicationPeer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20334 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927387/HBASE-20334.1.patch |
| Optional Tests | asflicense shellcheck shelldocs javac javadoc unit
shadedjars hadoopcheck xml compile |
| uname | Linux 214d3635f4e5 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 40f0a43462 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| shellcheck | v0.4.4 |
| shellcheck |
https://bu
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502155#comment-16502155
]
Josh Elser commented on HBASE-20334:
{quote}Yes, this is currently an end-to-end test. I could adapt it to
optionally use an existing cluster.
{quote}
My only concern was using a top-level directory in HDFS if it was a "secure"
instance. That would very likely require HDFS superuser access to create. If
we're using a minicluster only, that's perfectly fine (I think supporting a
real hdfs complicates this unnecessarily).
{quote}I could structure things differently by putting this test into its own
folder and including them as resources there. The down side is that the script
will no longer be self contained. At this point I don't have a strong feeling
either way.
{quote}
Yeah, I feel you. I didn't know of any bash "magic" that we could do which
would still give us a self-contained "unit" as you've created now. No strong
feelings from me either. I think what you have now is great.
> add a test that expressly uses both our shaded client and the one from hadoop
> 3
> ---
>
> Key: HBASE-20334
> URL: https://issues.apache.org/jira/browse/HBASE-20334
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20334.0.patch
>
>
> Since we're making a shaded client that bleed out of our namespace and into
> Hadoop's, we should ensure that we can show our clients coexisting. Even if
> it's just an IT that successfully talks to both us and HDFS via our
> respective shaded clients, that'd be a big help in keeping us proactive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502146#comment-16502146
]
Sean Busbey commented on HBASE-20334:
-
{quote}
bq. /path/to/component/bin-install
You mean the HBase installation by this? (the unpacked HBase binary tarball)
{quote}
yes.
{quote}
{code}
+
+hbase.rootdir
+
+/hbase
+
{code}
Double-checking, the script is always starting up a minicluster – never trying
to use an existing HDFS installation?
{quote}
Yes, this is currently an end-to-end test. I could adapt it to optionally use
an existing cluster.
{quote}
{code}
+yarn_server_tests_test_jar="$3"
+mapred_jobclient_test_jar="$4"
{code}
Could add some bash "is file" conditional checks on these two.
{quote}
will do.
{quote}
I'm sure you would've done this on your own already if there was an easy way,
but is there a way we could avoid generated the TSV data and the Java file at
the end of the script?
{quote}
I could structure things differently by putting this test into its own folder
and including them as resources there. The down side is that the script will no
longer be self contained. At this point I don't have a strong feeling either
way.
> add a test that expressly uses both our shaded client and the one from hadoop
> 3
> ---
>
> Key: HBASE-20334
> URL: https://issues.apache.org/jira/browse/HBASE-20334
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20334.0.patch
>
>
> Since we're making a shaded client that bleed out of our namespace and into
> Hadoop's, we should ensure that we can show our clients coexisting. Even if
> it's just an IT that successfully talks to both us and HDFS via our
> respective shaded clients, that'd be a big help in keeping us proactive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502135#comment-16502135
]
Josh Elser commented on HBASE-20334:
{code:java}
/path/to/component/bin-install{code}
You mean the HBase installation by this? (the unpacked HBase binary tarball)
{code:java}
+
+hbase.rootdir
+
+/hbase
+ {code}
Double-checking, the script is always starting up a minicluster – never trying
to use an existing HDFS installation?
{code:java}
+yarn_server_tests_test_jar="$3"
+mapred_jobclient_test_jar="$4"{code}
Could add some bash "is file" conditional checks on these two.
I'm sure you would've done this on your own already if there was an easy way,
but is there a way we could avoid generated the TSV data and the Java file at
the end of the script?
> add a test that expressly uses both our shaded client and the one from hadoop
> 3
> ---
>
> Key: HBASE-20334
> URL: https://issues.apache.org/jira/browse/HBASE-20334
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20334.0.patch
>
>
> Since we're making a shaded client that bleed out of our namespace and into
> Hadoop's, we should ensure that we can show our clients coexisting. Even if
> it's just an IT that successfully talks to both us and HDFS via our
> respective shaded clients, that'd be a big help in keeping us proactive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16484645#comment-16484645 ] Sean Busbey commented on HBASE-20334: - bq -1 shellcheck 0m 2s The patch generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2) interesting. this is reading in a heredoc, mistakenly. I'll see if I can ignore it. bq. -1 whitespace 0m 0s The patch has 29 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply bq. -1 whitespace 0m 1s The patch 48 line(s) with tabs. This is the TSV file the test uses with the importtsv MR job. I'll see if I can ignore it. > add a test that expressly uses both our shaded client and the one from hadoop > 3 > --- > > Key: HBASE-20334 > URL: https://issues.apache.org/jira/browse/HBASE-20334 > Project: HBase > Issue Type: Sub-task > Components: hadoop3, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Attachments: HBASE-20334.0.patch > > > Since we're making a shaded client that bleed out of our namespace and into > Hadoop's, we should ensure that we can show our clients coexisting. Even if > it's just an IT that successfully talks to both us and HDFS via our > respective shaded clients, that'd be a big help in keeping us proactive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16484610#comment-16484610
]
Hadoop QA commented on HBASE-20334:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
3s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:red}-1{color} | {color:red} hbaseanti {color} | {color:red} 0m
0s{color} | {color:red} The patch appears use HTrace 3 classes instead of
HTrace 4. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
3s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hbase-checkstyle hbase-testing-util hbase-assembly hbase-shaded
hbase-shaded/hbase-shaded-client hbase-shaded/hbase-shaded-mapreduce
hbase-shaded/hbase-shaded-check-invariants . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
34s{color} | {color:green} root: The patch generated 0 new + 3 unchanged - 1
fixed = 3 total (was 4) {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
2s{color} | {color:red} The patch generated 1 new + 2 unchanged - 0 fixed = 3
total (was 2) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 29 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch 48 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
31s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
33s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 21s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hbase-checkstyle hbase-testing-util hbase-shaded
hbase-shaded/hbase-shaded-client hbase-shaded/hbase-shaded-client-byo-hadoop
hbase-shaded/hbase-shaded-mapreduce . hbase-assembly
hbase-shaded/hbase-shaded-check-invariants
hbase-shaded/hbase-shaded-with-hadoop-check-invariants {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 11m
1s{color} | {color:green} the patch passed {color} |
| {color:
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16484265#comment-16484265 ] Hadoop QA commented on HBASE-20334: --- (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HBASE-Build/12901/console in case of problems. > add a test that expressly uses both our shaded client and the one from hadoop > 3 > --- > > Key: HBASE-20334 > URL: https://issues.apache.org/jira/browse/HBASE-20334 > Project: HBase > Issue Type: Sub-task > Components: hadoop3, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Attachments: HBASE-20334.0.patch > > > Since we're making a shaded client that bleed out of our namespace and into > Hadoop's, we should ensure that we can show our clients coexisting. Even if > it's just an IT that successfully talks to both us and HDFS via our > respective shaded clients, that'd be a big help in keeping us proactive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453175#comment-16453175 ] Sean Busbey commented on HBASE-20334: - here's my plan: * start single node hadoop cluster via CLI Minicluster ([docs for 2.7|http://hadoop.apache.org/docs/r2.7.6/hadoop-project-dist/hadoop-common/CLIMiniCluster.html], [docs for 3.0.1|http://hadoop.apache.org/docs/r3.0.2/hadoop-project-dist/hadoop-common/CLIMiniCluster.html]) * start HBase [Standalone-over-HDFS mode|http://hbase.apache.org/book.html#standalone.over.hdfs] * load example TSV file to HDFS via Hadoop CLI * import from tsv using shaded mapreduce * use utility program to scan result of import and compare it to data in HDFS, using shaded hbase client and hadoop dependencies (shaded hadoop client in the case of Hadoop 3) two questions that I'd like feedback on, but will just pick something if needed: a) where does the utility program live? is it in our code repo? do I generate it in the test? hbase-downstreamer? b) where does this test go? I could make it a yetus plugin. that would let us choose running it in precommit in addition to nightly if we wanted. Or I could just add it as a non-yetus step to our nightly builds, ala the "check source artifact" one. > add a test that expressly uses both our shaded client and the one from hadoop > 3 > --- > > Key: HBASE-20334 > URL: https://issues.apache.org/jira/browse/HBASE-20334 > Project: HBase > Issue Type: Sub-task > Components: hadoop3, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > Since we're making a shaded client that bleed out of our namespace and into > Hadoop's, we should ensure that we can show our clients coexisting. Even if > it's just an IT that successfully talks to both us and HDFS via our > respective shaded clients, that'd be a big help in keeping us proactive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
