[jira] [Commented] (HDFS-11668) Ozone: misc improvements for SCM CLI
[
https://issues.apache.org/jira/browse/HDFS-11668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16455041#comment-16455041
]
Hudson commented on HDFS-11668:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14070 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14070/])
HDFS-11668. Ozone: misc improvements for SCM CLI. Contributed by Weiwei
(omalley: rev 1a245accb5f83729e1b02c3f2f0e6e0feb79c04a)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/container/common/TestDatanodeStateMachine.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/states/datanode/InitDatanodeState.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/cli/container/CreateContainerHandler.java
> Ozone: misc improvements for SCM CLI
>
>
> Key: HDFS-11668
> URL: https://issues.apache.org/jira/browse/HDFS-11668
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Labels: cli, command-line
> Attachments: HDFS-11668-HDFS-7240.001.patch,
> HDFS-11668-HDFS-7240.002.patch, HDFS-11668-HDFS-7240.003.patch,
> HDFS-11668-HDFS-7240.004.patch
>
>
> Once HDFS-11649 is done, there are some improvements need to be done in order
> to make SCM CLI better to use in a pseudo cluster, this includes
> # HDFS-11649 adds java classes for CLIs, we will need to add shell code to
> expose these commands
> # Better error messages when missing some key ozone configurations, e.g
> {{ozone.scm.names}}, {{ozone.scm.datanode.id}} ... etc
> # Better logging. Currently {{DatanodeStateMachine}} prints very limited
> logs, adds some more logs to indicate state transition is necessary.
> The ultimate goal of this ticket is to ensure SCM CLI can work nicely on a
> pseudo cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11668) Ozone: misc improvements for SCM CLI
[
https://issues.apache.org/jira/browse/HDFS-11668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16450632#comment-16450632
]
Hudson commented on HDFS-11668:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14057 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14057/])
HDFS-11668. Ozone: misc improvements for SCM CLI. Contributed by Weiwei (xyao:
rev a2c6f594baab7357cf83749986c31d07acbb1980)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/states/datanode/InitDatanodeState.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/container/common/TestDatanodeStateMachine.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/cli/container/CreateContainerHandler.java
> Ozone: misc improvements for SCM CLI
>
>
> Key: HDFS-11668
> URL: https://issues.apache.org/jira/browse/HDFS-11668
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Labels: cli, command-line
> Attachments: HDFS-11668-HDFS-7240.001.patch,
> HDFS-11668-HDFS-7240.002.patch, HDFS-11668-HDFS-7240.003.patch,
> HDFS-11668-HDFS-7240.004.patch
>
>
> Once HDFS-11649 is done, there are some improvements need to be done in order
> to make SCM CLI better to use in a pseudo cluster, this includes
> # HDFS-11649 adds java classes for CLIs, we will need to add shell code to
> expose these commands
> # Better error messages when missing some key ozone configurations, e.g
> {{ozone.scm.names}}, {{ozone.scm.datanode.id}} ... etc
> # Better logging. Currently {{DatanodeStateMachine}} prints very limited
> logs, adds some more logs to indicate state transition is necessary.
> The ultimate goal of this ticket is to ensure SCM CLI can work nicely on a
> pseudo cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11668) Ozone: misc improvements for SCM CLI
[
https://issues.apache.org/jira/browse/HDFS-11668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977216#comment-15977216
]
Xiaoyu Yao commented on HDFS-11668:
---
Thanks [~cheersyang] for the update. +1 for v4 patch and I will commit it
shortly.
> Ozone: misc improvements for SCM CLI
>
>
> Key: HDFS-11668
> URL: https://issues.apache.org/jira/browse/HDFS-11668
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Labels: cli, command-line
> Attachments: HDFS-11668-HDFS-7240.001.patch,
> HDFS-11668-HDFS-7240.002.patch, HDFS-11668-HDFS-7240.003.patch,
> HDFS-11668-HDFS-7240.004.patch
>
>
> Once HDFS-11649 is done, there are some improvements need to be done in order
> to make SCM CLI better to use in a pseudo cluster, this includes
> # HDFS-11649 adds java classes for CLIs, we will need to add shell code to
> expose these commands
> # Better error messages when missing some key ozone configurations, e.g
> {{ozone.scm.names}}, {{ozone.scm.datanode.id}} ... etc
> # Better logging. Currently {{DatanodeStateMachine}} prints very limited
> logs, adds some more logs to indicate state transition is necessary.
> The ultimate goal of this ticket is to ensure SCM CLI can work nicely on a
> pseudo cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11668) Ozone: misc improvements for SCM CLI
[
https://issues.apache.org/jira/browse/HDFS-11668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976259#comment-15976259
]
Hadoop QA commented on HDFS-11668:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
7s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
6s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 2m
36s{color} | {color:red} The patch generated 1 new + 74 unchanged - 2 fixed =
75 total (was 76) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 40s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.ozone.scm.node.TestContainerPlacement |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | HDFS-11668 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12864203/HDFS-11668-HDFS-7240.004.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs compile
javac javadoc mvninstall findbugs checkstyle |
| uname | Linux 57eac9a8ad19 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 8ec13a3 |
| Default Java | 1.8.0_121 |
| shellcheck | v0.4.6 |
| findbugs | v3.0.0 |
| shellcheck |
https://builds.apache.org/job/PreCommit-HDFS-Build/19158/artifact/patchprocess/diff-patch-shellcheck.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19158/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/1
[jira] [Commented] (HDFS-11668) Ozone: misc improvements for SCM CLI
[
https://issues.apache.org/jira/browse/HDFS-11668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976146#comment-15976146
]
Weiwei Yang commented on HDFS-11668:
Sure [~xyao], added the check in v4 patch, thank you!
> Ozone: misc improvements for SCM CLI
>
>
> Key: HDFS-11668
> URL: https://issues.apache.org/jira/browse/HDFS-11668
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Labels: cli, command-line
> Attachments: HDFS-11668-HDFS-7240.001.patch,
> HDFS-11668-HDFS-7240.002.patch, HDFS-11668-HDFS-7240.003.patch,
> HDFS-11668-HDFS-7240.004.patch
>
>
> Once HDFS-11649 is done, there are some improvements need to be done in order
> to make SCM CLI better to use in a pseudo cluster, this includes
> # HDFS-11649 adds java classes for CLIs, we will need to add shell code to
> expose these commands
> # Better error messages when missing some key ozone configurations, e.g
> {{ozone.scm.names}}, {{ozone.scm.datanode.id}} ... etc
> # Better logging. Currently {{DatanodeStateMachine}} prints very limited
> logs, adds some more logs to indicate state transition is necessary.
> The ultimate goal of this ticket is to ensure SCM CLI can work nicely on a
> pseudo cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11668) Ozone: misc improvements for SCM CLI
[
https://issues.apache.org/jira/browse/HDFS-11668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976140#comment-15976140
]
Xiaoyu Yao commented on HDFS-11668:
---
Can we guard the {{LOG.debug(...}} with {{if (LOG.isDebugEnabled())}} since
some of the parameters involve method calls?
+1 after that.
> Ozone: misc improvements for SCM CLI
>
>
> Key: HDFS-11668
> URL: https://issues.apache.org/jira/browse/HDFS-11668
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Labels: cli, command-line
> Attachments: HDFS-11668-HDFS-7240.001.patch,
> HDFS-11668-HDFS-7240.002.patch, HDFS-11668-HDFS-7240.003.patch
>
>
> Once HDFS-11649 is done, there are some improvements need to be done in order
> to make SCM CLI better to use in a pseudo cluster, this includes
> # HDFS-11649 adds java classes for CLIs, we will need to add shell code to
> expose these commands
> # Better error messages when missing some key ozone configurations, e.g
> {{ozone.scm.names}}, {{ozone.scm.datanode.id}} ... etc
> # Better logging. Currently {{DatanodeStateMachine}} prints very limited
> logs, adds some more logs to indicate state transition is necessary.
> The ultimate goal of this ticket is to ensure SCM CLI can work nicely on a
> pseudo cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11668) Ozone: misc improvements for SCM CLI
[
https://issues.apache.org/jira/browse/HDFS-11668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976125#comment-15976125
]
Hadoop QA commented on HDFS-11668:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
7s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
27s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 2m
42s{color} | {color:red} The patch generated 1 new + 74 unchanged - 2 fixed =
75 total (was 76) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 20s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | HDFS-11668 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12864193/HDFS-11668-HDFS-7240.003.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs compile
javac javadoc mvninstall findbugs checkstyle |
| uname | Linux 3645283f2b22 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 0bf416a |
| Default Java | 1.8.0_121 |
| shellcheck | v0.4.6 |
| findbugs | v3.0.0 |
| shellcheck |
https://builds.apache.org/job/PreCommit-HDFS-Build/19155/artifact/patchprocess/diff-patch-shellcheck.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19155/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19155/testReport/ |
| modules |
[jira] [Commented] (HDFS-11668) Ozone: misc improvements for SCM CLI
[
https://issues.apache.org/jira/browse/HDFS-11668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976024#comment-15976024
]
Weiwei Yang commented on HDFS-11668:
Sure [~xyao], thanks for the review, I just updated the log level from info to
debug in v3 patch.
> Ozone: misc improvements for SCM CLI
>
>
> Key: HDFS-11668
> URL: https://issues.apache.org/jira/browse/HDFS-11668
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Labels: cli, command-line
> Attachments: HDFS-11668-HDFS-7240.001.patch,
> HDFS-11668-HDFS-7240.002.patch, HDFS-11668-HDFS-7240.003.patch
>
>
> Once HDFS-11649 is done, there are some improvements need to be done in order
> to make SCM CLI better to use in a pseudo cluster, this includes
> # HDFS-11649 adds java classes for CLIs, we will need to add shell code to
> expose these commands
> # Better error messages when missing some key ozone configurations, e.g
> {{ozone.scm.names}}, {{ozone.scm.datanode.id}} ... etc
> # Better logging. Currently {{DatanodeStateMachine}} prints very limited
> logs, adds some more logs to indicate state transition is necessary.
> The ultimate goal of this ticket is to ensure SCM CLI can work nicely on a
> pseudo cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11668) Ozone: misc improvements for SCM CLI
[
https://issues.apache.org/jira/browse/HDFS-11668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975317#comment-15975317
]
Xiaoyu Yao commented on HDFS-11668:
---
Thanks [~cheersyang] for working on this. The patch looks good to me. I only
have one minor issue.
Can we change the log level from info to debug in StateContext.java Line 194?
> Ozone: misc improvements for SCM CLI
>
>
> Key: HDFS-11668
> URL: https://issues.apache.org/jira/browse/HDFS-11668
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Labels: cli, command-line
> Attachments: HDFS-11668-HDFS-7240.001.patch,
> HDFS-11668-HDFS-7240.002.patch
>
>
> Once HDFS-11649 is done, there are some improvements need to be done in order
> to make SCM CLI better to use in a pseudo cluster, this includes
> # HDFS-11649 adds java classes for CLIs, we will need to add shell code to
> expose these commands
> # Better error messages when missing some key ozone configurations, e.g
> {{ozone.scm.names}}, {{ozone.scm.datanode.id}} ... etc
> # Better logging. Currently {{DatanodeStateMachine}} prints very limited
> logs, adds some more logs to indicate state transition is necessary.
> The ultimate goal of this ticket is to ensure SCM CLI can work nicely on a
> pseudo cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11668) Ozone: misc improvements for SCM CLI
[
https://issues.apache.org/jira/browse/HDFS-11668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974785#comment-15974785
]
Hadoop QA commented on HDFS-11668:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
9s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 2m
56s{color} | {color:red} The patch generated 1 new + 74 unchanged - 2 fixed =
75 total (was 76) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.cblock.TestCBlockCLI |
| | hadoop.cblock.TestCBlockServerPersistence |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestBuckets |
| | org.apache.hadoop.ozone.web.client.TestVolume |
| | org.apache.hadoop.ozone.web.TestOzoneRestWithMiniCluster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | HDFS-11668 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12864002/HDFS-11668-HDFS-7240.002.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs compile
javac javadoc mvninstall findbugs checkstyle |
| uname | Linux 03767c67df05 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 0bf416a |
| Default Java | 1.8.0_121 |
| shellcheck | v0.4.6 |
| findbugs | v3.0.0 |
| shellcheck |
https://builds.apache.org/job/PreCommit-HDFS-Build/19144/artifact/patchprocess/diff-patch-shellcheck.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19144/artifact/patchprocess/patch-unit-hadoop-hdfs-proj
[jira] [Commented] (HDFS-11668) Ozone: misc improvements for SCM CLI
[
https://issues.apache.org/jira/browse/HDFS-11668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974648#comment-15974648
]
Hadoop QA commented on HDFS-11668:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
8s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 2m
38s{color} | {color:red} The patch generated 1 new + 74 unchanged - 2 fixed =
75 total (was 76) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 22s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.ozone.scm.node.TestContainerPlacement |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | HDFS-11668 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863994/HDFS-11668-HDFS-7240.001.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs compile
javac javadoc mvninstall findbugs checkstyle |
| uname | Linux 0aeb11df1c8d 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 0bf416a |
| Default Java | 1.8.0_121 |
| shellcheck | v0.4.6 |
| findbugs | v3.0.0 |
| shellcheck |
https://builds.apache.org/job/PreCommit-HDFS-Build/19143/artifact/patchprocess/diff-patch-shellcheck.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19143/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19143/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-
[jira] [Commented] (HDFS-11668) Ozone: misc improvements for SCM CLI
[
https://issues.apache.org/jira/browse/HDFS-11668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974550#comment-15974550
]
Weiwei Yang commented on HDFS-11668:
Hi [~anu], [~xyao]
Please help to review this JIRA, I am trying to get SCM CLI work nicely on a
pseudo cluster, during my testing, I found the problems I wrote in the
description. This patch includes following fixes
*hdfs*
Integrate SCM CLI into hdfs script, this is the followup work after HDFS-11649.
With this patch, {{bin/hdfs scm}} will start scm daemon, and if there is more
arguments given, e.g {{bin/hdfs scm -container}}, it will parse to SCM CLI
classes.
*InitDatanodeState*
Make sure datanode id file (ozone.scm.datanode.id) is properly set in
configuration, otherwise set next datanode state to SHUTDOWN. And it prompts
user a meaningful error message "A valid file path is needed for config setting
ozone.scm.datanode.id" to help user understand what was wrong.
*StateContext*
Added a log message to indicate how datanode state is transited, log looks like
{noformat}
17/04/19 04:57:33 INFO statemachine.StateContext: Task InitDatanodeState
executed, state transited from INIT to RUNNING
{noformat}
this log is only printed when state changes.
*TestDatanodeStateMachine*
Renamed test case {{testDatanodeStateMachineWithInvalidConfiguration}} and
added test coverage for a negative case that {{ozone.scm.datanode.id}} was not
set.
I tested this on my single node cluster, with this patch, I am able to create a
container with following command
{noformat}
[wwei@ozone1 hadoop-3.0.0-alpha3-SNAPSHOT]$ ./bin/hdfs scm -container -create
-p 20170418c1
17/04/19 05:18:10 INFO cli.OzoneCommandHandler: Create container : 20170418c1
17/04/19 05:18:10 INFO client.ContainerOperationClient: Created container
20170418c1 leader:172.16.165.133:9866 machines:[172.16.165.133:9866]
[wwei@ozone1 hadoop-3.0.0-alpha3-SNAPSHOT]$ ./bin/hdfs scm -container -create
-p 20170418c1
17/04/19 05:18:27 INFO cli.OzoneCommandHandler: Create container :20170418c1
Error executing
command:org.apache.hadoop.ipc.RemoteException(java.io.IOException): Specified
container already exists. key : 20170418c1
...
{noformat}
first operation succeed and second failed because container already exists.
Please kindly help to review, thanks.
> Ozone: misc improvements for SCM CLI
>
>
> Key: HDFS-11668
> URL: https://issues.apache.org/jira/browse/HDFS-11668
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Labels: cli, command-line
> Attachments: HDFS-11668-HDFS-7240.001.patch
>
>
> Once HDFS-11649 is done, there are some improvements need to be done in order
> to make SCM CLI better to use in a pseudo cluster, this includes
> # HDFS-11649 adds java classes for CLIs, we will need to add shell code to
> expose these commands
> # Better error messages when missing some key ozone configurations, e.g
> {{ozone.scm.names}}, {{ozone.scm.datanode.id}} ... etc
> # Better logging. Currently {{DatanodeStateMachine}} prints very limited
> logs, adds some more logs to indicate state transition is necessary.
> The ultimate goal of this ticket is to ensure SCM CLI can work nicely on a
> pseudo cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
