[jira] [Commented] (HDFS-12967) NNBench should support multi-cluster access
[ https://issues.apache.org/jira/browse/HDFS-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895231#comment-16895231 ] Hudson commented on HDFS-12967: --- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17001 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17001/]) HDFS-12967. NNBench should support multi-cluster access. Contributed by (weichiu: rev 8a59cd1b8a2509ea03200e3617775b8c853d4bf2) * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/TestNNBench.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java > NNBench should support multi-cluster access > --- > > Key: HDFS-12967 > URL: https://issues.apache.org/jira/browse/HDFS-12967 > Project: Hadoop HDFS > Issue Type: Improvement > Components: benchmarks >Reporter: Chen Zhang >Assignee: Chen Zhang >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-12967-001.patch, HDFS-12967-002.patch, > HDFS-12967-003.patch, HDFS-12967-004.patch > > > Sometimes we need to run NNBench for some scaling tests after made some > improvements on NameNode, so we have to deploy a new HDFS cluster and a new > Yarn cluster. > If NNBench support multi-cluster access, we only need to deploy a new HDFS > test cluster and add it to existing YARN cluster, it'll make the scaling test > easier. > Even more, if we want to do some A-B test, we have to run NNBench on > different HDFS clusters, this patch will be helpful. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12967) NNBench should support multi-cluster access
[ https://issues.apache.org/jira/browse/HDFS-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894253#comment-16894253 ] Chen Zhang commented on HDFS-12967: --- Thanks [~jojochuang], my email is chzhang1...@gmail.com > NNBench should support multi-cluster access > --- > > Key: HDFS-12967 > URL: https://issues.apache.org/jira/browse/HDFS-12967 > Project: Hadoop HDFS > Issue Type: Improvement > Components: benchmarks >Reporter: Chen Zhang >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-12967-001.patch, HDFS-12967-002.patch, > HDFS-12967-003.patch, HDFS-12967-004.patch > > > Sometimes we need to run NNBench for some scaling tests after made some > improvements on NameNode, so we have to deploy a new HDFS cluster and a new > Yarn cluster. > If NNBench support multi-cluster access, we only need to deploy a new HDFS > test cluster and add it to existing YARN cluster, it'll make the scaling test > easier. > Even more, if we want to do some A-B test, we have to run NNBench on > different HDFS clusters, this patch will be helpful. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12967) NNBench should support multi-cluster access
[ https://issues.apache.org/jira/browse/HDFS-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894203#comment-16894203 ] Wei-Chiu Chuang commented on HDFS-12967: [~zhangchen] do you have an email address? I'd like to credit you in the git commit message. Thanks. > NNBench should support multi-cluster access > --- > > Key: HDFS-12967 > URL: https://issues.apache.org/jira/browse/HDFS-12967 > Project: Hadoop HDFS > Issue Type: Improvement > Components: benchmarks >Reporter: Chen Zhang >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-12967-001.patch, HDFS-12967-002.patch, > HDFS-12967-003.patch, HDFS-12967-004.patch > > > Sometimes we need to run NNBench for some scaling tests after made some > improvements on NameNode, so we have to deploy a new HDFS cluster and a new > Yarn cluster. > If NNBench support multi-cluster access, we only need to deploy a new HDFS > test cluster and add it to existing YARN cluster, it'll make the scaling test > easier. > Even more, if we want to do some A-B test, we have to run NNBench on > different HDFS clusters, this patch will be helpful. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12967) NNBench should support multi-cluster access
[ https://issues.apache.org/jira/browse/HDFS-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894200#comment-16894200 ] Wei-Chiu Chuang commented on HDFS-12967: +1 > NNBench should support multi-cluster access > --- > > Key: HDFS-12967 > URL: https://issues.apache.org/jira/browse/HDFS-12967 > Project: Hadoop HDFS > Issue Type: Improvement > Components: benchmarks >Reporter: Chen Zhang >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-12967-001.patch, HDFS-12967-002.patch, > HDFS-12967-003.patch, HDFS-12967-004.patch > > > Sometimes we need to run NNBench for some scaling tests after made some > improvements on NameNode, so we have to deploy a new HDFS cluster and a new > Yarn cluster. > If NNBench support multi-cluster access, we only need to deploy a new HDFS > test cluster and add it to existing YARN cluster, it'll make the scaling test > easier. > Even more, if we want to do some A-B test, we have to run NNBench on > different HDFS clusters, this patch will be helpful. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12967) NNBench should support multi-cluster access
[ https://issues.apache.org/jira/browse/HDFS-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873174#comment-16873174 ] Chen Zhang commented on HDFS-12967: --- Hi [~jojochuang], thanks for review. I've fixed these checkstyle errors, could you help to review the patch again? Thanks > NNBench should support multi-cluster access > --- > > Key: HDFS-12967 > URL: https://issues.apache.org/jira/browse/HDFS-12967 > Project: Hadoop HDFS > Issue Type: Improvement > Components: benchmarks >Reporter: Chen Zhang >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-12967-001.patch, HDFS-12967-002.patch, > HDFS-12967-003.patch, HDFS-12967-004.patch > > > Sometimes we need to run NNBench for some scaling tests after made some > improvements on NameNode, so we have to deploy a new HDFS cluster and a new > Yarn cluster. > If NNBench support multi-cluster access, we only need to deploy a new HDFS > test cluster and add it to existing YARN cluster, it'll make the scaling test > easier. > Even more, if we want to do some A-B test, we have to run NNBench on > different HDFS clusters, this patch will be helpful. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12967) NNBench should support multi-cluster access
[
https://issues.apache.org/jira/browse/HDFS-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873137#comment-16873137
]
Hadoop QA commented on HDFS-12967:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
The patch generated 0 new + 145 unchanged - 2 fixed = 145 total (was 147)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}123m 54s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
29s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}168m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-12967 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12972930/HDFS-12967-004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 0b1afc87bd92 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a79bdf7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/27086/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/27086/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HDFS-Build/27086/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 1281 (vs. ulimit of 1) |
| modules | C:
[jira] [Commented] (HDFS-12967) NNBench should support multi-cluster access
[
https://issues.apache.org/jira/browse/HDFS-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866312#comment-16866312
]
Wei-Chiu Chuang commented on HDFS-12967:
{quote}./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java:196:
"Supports cross-cluster access by using full path with schema and
cluster. " +: '"Supports cross-cluster access by using full path with schema
and cluster. "' has incorrect indentation level 6, expected level should be 8.
[Indentation]
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java:196:
"Supports cross-cluster access by using full path with schema and
cluster. " +: Line is longer than 80 characters (found 84). [LineLength]
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/TestNNBench.java:94:
String[] genArgs = { "-operation", operation, "-baseDir", baseDir,:24: '{'
is followed by whitespace. [NoWhitespaceAfter]
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/TestNNBench.java:95:
"-startTime", "" + (Time.now() / 1000 + 3) , "-blockSize", "1024"};:52:
',' is preceded with whitespace. [NoWhitespaceBefore]
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/TestNNBench.java:100:
private void runNNBench(Configuration conf, String operation) throws
Exception {: Line is longer than 80 characters (found 82). [LineLength]{quote}
> NNBench should support multi-cluster access
> ---
>
> Key: HDFS-12967
> URL: https://issues.apache.org/jira/browse/HDFS-12967
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: benchmarks
>Reporter: Chen Zhang
>Assignee: Chen Zhang
>Priority: Major
> Attachments: HDFS-12967-001.patch, HDFS-12967-002.patch,
> HDFS-12967-003.patch
>
>
> Sometimes we need to run NNBench for some scaling tests after made some
> improvements on NameNode, so we have to deploy a new HDFS cluster and a new
> Yarn cluster.
> If NNBench support multi-cluster access, we only need to deploy a new HDFS
> test cluster and add it to existing YARN cluster, it'll make the scaling test
> easier.
> Even more, if we want to do some A-B test, we have to run NNBench on
> different HDFS clusters, this patch will be helpful.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12967) NNBench should support multi-cluster access
[
https://issues.apache.org/jira/browse/HDFS-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866105#comment-16866105
]
Hadoop QA commented on HDFS-12967:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 21s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
The patch generated 5 new + 146 unchanged - 1 fixed = 151 total (was 147)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}150m 28s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
48s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}204m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-12967 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12962062/HDFS-12967-003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 09aa9fccd541 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3d020e9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26976/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26976/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26976/testReport/ |
|
[jira] [Commented] (HDFS-12967) NNBench should support multi-cluster access
[ https://issues.apache.org/jira/browse/HDFS-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866004#comment-16866004 ] Wei-Chiu Chuang commented on HDFS-12967: LGTM other than the checkstyle warnings. Sorry I missed your message. Re-trigger precommit to see the checkstyle warning > NNBench should support multi-cluster access > --- > > Key: HDFS-12967 > URL: https://issues.apache.org/jira/browse/HDFS-12967 > Project: Hadoop HDFS > Issue Type: Improvement > Components: benchmarks >Reporter: Chen Zhang >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-12967-001.patch, HDFS-12967-002.patch, > HDFS-12967-003.patch > > > Sometimes we need to run NNBench for some scaling tests after made some > improvements on NameNode, so we have to deploy a new HDFS cluster and a new > Yarn cluster. > If NNBench support multi-cluster access, we only need to deploy a new HDFS > test cluster and add it to existing YARN cluster, it'll make the scaling test > easier. > Even more, if we want to do some A-B test, we have to run NNBench on > different HDFS clusters, this patch will be helpful. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12967) NNBench should support multi-cluster access
[
https://issues.apache.org/jira/browse/HDFS-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790314#comment-16790314
]
Hadoop QA commented on HDFS-12967:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 8m
6s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
The patch generated 5 new + 146 unchanged - 1 fixed = 151 total (was 147)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}141m 52s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
33s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}200m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapred.TestMRTimelineEventHandling |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-12967 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12962062/HDFS-12967-003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9ca1155446e6 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c730786 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26453/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| unit |
[jira] [Commented] (HDFS-12967) NNBench should support multi-cluster access
[
https://issues.apache.org/jira/browse/HDFS-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790241#comment-16790241
]
Hadoop QA commented on HDFS-12967:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
The patch generated 3 new + 146 unchanged - 1 fixed = 149 total (was 147)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}129m 10s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
42s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}196m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapred.TestMRTimelineEventHandling |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-12967 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12961949/HDFS-12967-002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e134b492ee25 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2a54fea |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26451/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26451/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
[jira] [Commented] (HDFS-12967) NNBench should support multi-cluster access
[
https://issues.apache.org/jira/browse/HDFS-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790200#comment-16790200
]
Hadoop QA commented on HDFS-12967:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 8m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
20s{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 20s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 17s{color} | {color:orange} The patch fails to run checkstyle in
hadoop-mapreduce-client-jobclient {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
19s{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 3m
41s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
18s{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 21s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-12967 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12962059/HDFS-12967-002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 22734faacc70 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2a54fea |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/26452/artifact/out/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/26452/artifact/out/patch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| javac |
[jira] [Commented] (HDFS-12967) NNBench should support multi-cluster access
[ https://issues.apache.org/jira/browse/HDFS-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790168#comment-16790168 ] Chen Zhang commented on HDFS-12967: --- [~jojochuang], would you please help me to review this patch again? I am sorry that it has been delayed for so long. > NNBench should support multi-cluster access > --- > > Key: HDFS-12967 > URL: https://issues.apache.org/jira/browse/HDFS-12967 > Project: Hadoop HDFS > Issue Type: Improvement > Components: benchmarks >Reporter: Chen Zhang >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-12967-001.patch, HDFS-12967-002.patch > > > Sometimes we need to run NNBench for some scaling tests after made some > improvements on NameNode, so we have to deploy a new HDFS cluster and a new > Yarn cluster. > If NNBench support multi-cluster access, we only need to deploy a new HDFS > test cluster and add it to existing YARN cluster, it'll make the scaling test > easier. > Even more, if we want to do some A-B test, we have to run NNBench on > different HDFS clusters, this patch will be helpful. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12967) NNBench should support multi-cluster access
[
https://issues.apache.org/jira/browse/HDFS-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790147#comment-16790147
]
Chen Zhang commented on HDFS-12967:
---
{quote}Did you try {{TestDFSIO}} instead. Should be the right tool for scale
testing.
{quote}
Thanks [~shv] for your advice, we used TestDFSIO to measure the cluster
throughput and performance, I think it's more suitable for data access
performance testing，NNBench, as it's name implies, is special designed for
stress test of NameNode operations
{quote}Also NNBench has {{-baseDir}} option.
{quote}
My patch is just some minor changes on this option to make NNBench support
multi-cluster
> NNBench should support multi-cluster access
> ---
>
> Key: HDFS-12967
> URL: https://issues.apache.org/jira/browse/HDFS-12967
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: benchmarks
>Reporter: Chen Zhang
>Assignee: Chen Zhang
>Priority: Major
> Attachments: HDFS-12967-001.patch, HDFS-12967-002.patch
>
>
> Sometimes we need to run NNBench for some scaling tests after made some
> improvements on NameNode, so we have to deploy a new HDFS cluster and a new
> Yarn cluster.
> If NNBench support multi-cluster access, we only need to deploy a new HDFS
> test cluster and add it to existing YARN cluster, it'll make the scaling test
> easier.
> Even more, if we want to do some A-B test, we have to run NNBench on
> different HDFS clusters, this patch will be helpful.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12967) NNBench should support multi-cluster access
[ https://issues.apache.org/jira/browse/HDFS-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351273#comment-16351273 ] caixiaofeng commented on HDFS-12967: mark, it looks like a good function. > NNBench should support multi-cluster access > --- > > Key: HDFS-12967 > URL: https://issues.apache.org/jira/browse/HDFS-12967 > Project: Hadoop HDFS > Issue Type: Improvement > Components: benchmarks >Reporter: Chen Zhang >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-12967-001.patch > > > Sometimes we need to run NNBench for some scaling tests after made some > improvements on NameNode, so we have to deploy a new HDFS cluster and a new > Yarn cluster. > If NNBench support multi-cluster access, we only need to deploy a new HDFS > test cluster and add it to existing YARN cluster, it'll make the scaling test > easier. > Even more, if we want to do some A-B test, we have to run NNBench on > different HDFS clusters, this patch will be helpful. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12967) NNBench should support multi-cluster access
[
https://issues.apache.org/jira/browse/HDFS-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314899#comment-16314899
]
Konstantin Shvachko commented on HDFS-12967:
Did you try {{TestDFSIO}} instead. Should be the right tool for scale testing.
Also NNBench has {{-baseDir}} option.
> NNBench should support multi-cluster access
> ---
>
> Key: HDFS-12967
> URL: https://issues.apache.org/jira/browse/HDFS-12967
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: benchmarks
>Reporter: Chen Zhang
>Assignee: Chen Zhang
> Attachments: HDFS-12967-001.patch
>
>
> Sometimes we need to run NNBench for some scaling tests after made some
> improvements on NameNode, so we have to deploy a new HDFS cluster and a new
> Yarn cluster.
> If NNBench support multi-cluster access, we only need to deploy a new HDFS
> test cluster and add it to existing YARN cluster, it'll make the scaling test
> easier.
> Even more, if we want to do some A-B test, we have to run NNBench on
> different HDFS clusters, this patch will be helpful.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12967) NNBench should support multi-cluster access
[ https://issues.apache.org/jira/browse/HDFS-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305877#comment-16305877 ] Chen Zhang commented on HDFS-12967: --- Thanks for your response and suggestion, [~jojochuang]. bq. what about adding a new command parameter so that this support is visible I think it don't need a new command parameter, just using path with prefix like hdfs://some-cluster/user/foo will work. And it's reasonable to add some explanation in help text. bq. You should also consider adding tests in TestNNBench Thanks for reminding this, I'll add a test soon > NNBench should support multi-cluster access > --- > > Key: HDFS-12967 > URL: https://issues.apache.org/jira/browse/HDFS-12967 > Project: Hadoop HDFS > Issue Type: Improvement > Components: benchmarks >Reporter: Chen Zhang >Assignee: Chen Zhang > Attachments: HDFS-12967-001.patch > > > Sometimes we need to run NNBench for some scaling tests after made some > improvements on NameNode, so we have to deploy a new HDFS cluster and a new > Yarn cluster. > If NNBench support multi-cluster access, we only need to deploy a new HDFS > test cluster and add it to existing YARN cluster, it'll make the scaling test > easier. > Even more, if we want to do some A-B test, we have to run NNBench on > different HDFS clusters, this patch will be helpful. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12967) NNBench should support multi-cluster access
[ https://issues.apache.org/jira/browse/HDFS-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305635#comment-16305635 ] Wei-Chiu Chuang commented on HDFS-12967: Sounds a good improvement. For completeness, what about adding a new command parameter so that this support is visible? You should also consider adding tests in TestNNBench. Thanks > NNBench should support multi-cluster access > --- > > Key: HDFS-12967 > URL: https://issues.apache.org/jira/browse/HDFS-12967 > Project: Hadoop HDFS > Issue Type: Improvement > Components: benchmarks >Reporter: Chen Zhang >Assignee: Chen Zhang > Attachments: HDFS-12967-001.patch > > > Sometimes we need to run NNBench for some scaling tests after made some > improvements on NameNode, so we have to deploy a new HDFS cluster and a new > Yarn cluster. > If NNBench support multi-cluster access, we only need to deploy a new HDFS > test cluster and add it to existing YARN cluster, it'll make the scaling test > easier. > Even more, if we want to make some A-B test, we have to run NNBench on > different HDFS clusters, this patch will be helpful. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
