[jira] [Commented] (HDFS-11830) Ozone: Datanode needs to re-register to SCM if SCM is restarted
[ https://issues.apache.org/jira/browse/HDFS-11830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455072#comment-16455072 ] Hudson commented on HDFS-11830: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14070 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14070/]) HDFS-11830. Ozone: Datanode needs to re-register to SCM if SCM is (omalley: rev ae5242accbbd47e44adada58958ce7216ff092b3) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/StorageContainerDatanodeProtocol.proto * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/StorageContainerManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/node/SCMNodeManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/HeartbeatEndpointTask.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/protocol/commands/ReregisterCommand.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/node/CommandQueue.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/scm/node/TestNodeManager.java > Ozone: Datanode needs to re-register to SCM if SCM is restarted > --- > > Key: HDFS-11830 > URL: https://issues.apache.org/jira/browse/HDFS-11830 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Critical > Fix For: HDFS-7240 > > Attachments: HDFS-11830-HDFS-7240.001.patch, > HDFS-11830-HDFS-7240.002.patch, HDFS-11830-HDFS-7240.003.patch, > HDFS-11830-HDFS-7240.004.patch > > > Problem description: > # Start NN, DN, SCM > # Restart SCM and will see following warnings in SCM log > 17/05/02 00:47:08 WARN node.SCMNodeManager: SCM receive heartbeat from > unregistered datanode > Datanode could not re-establish communication with SCM afterwards. Propose to > fix this by adding a new command in HB handling telling datanode to > re-register with SCM. Datanode once received this command transits to > REGISTER state again to proceed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11830) Ozone: Datanode needs to re-register to SCM if SCM is restarted
[ https://issues.apache.org/jira/browse/HDFS-11830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450667#comment-16450667 ] Hudson commented on HDFS-11830: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14057 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14057/]) HDFS-11830. Ozone: Datanode needs to re-register to SCM if SCM is (wwei: rev b4e5c5543604ec9672cf445c6f22c2ca03de2d15) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/node/CommandQueue.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/StorageContainerManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/scm/node/TestNodeManager.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/protocol/commands/ReregisterCommand.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/HeartbeatEndpointTask.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/StorageContainerDatanodeProtocol.proto * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/node/SCMNodeManager.java > Ozone: Datanode needs to re-register to SCM if SCM is restarted > --- > > Key: HDFS-11830 > URL: https://issues.apache.org/jira/browse/HDFS-11830 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Critical > Fix For: HDFS-7240 > > Attachments: HDFS-11830-HDFS-7240.001.patch, > HDFS-11830-HDFS-7240.002.patch, HDFS-11830-HDFS-7240.003.patch, > HDFS-11830-HDFS-7240.004.patch > > > Problem description: > # Start NN, DN, SCM > # Restart SCM and will see following warnings in SCM log > 17/05/02 00:47:08 WARN node.SCMNodeManager: SCM receive heartbeat from > unregistered datanode > Datanode could not re-establish communication with SCM afterwards. Propose to > fix this by adding a new command in HB handling telling datanode to > re-register with SCM. Datanode once received this command transits to > REGISTER state again to proceed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11830) Ozone: Datanode needs to re-register to SCM if SCM is restarted
[ https://issues.apache.org/jira/browse/HDFS-11830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018252#comment-16018252 ] Weiwei Yang commented on HDFS-11830: Thanks [~msingh] for the review, I just committed this to the feature branch. > Ozone: Datanode needs to re-register to SCM if SCM is restarted > --- > > Key: HDFS-11830 > URL: https://issues.apache.org/jira/browse/HDFS-11830 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Critical > Fix For: HDFS-7240 > > Attachments: HDFS-11830-HDFS-7240.001.patch, > HDFS-11830-HDFS-7240.002.patch, HDFS-11830-HDFS-7240.003.patch, > HDFS-11830-HDFS-7240.004.patch > > > Problem description: > # Start NN, DN, SCM > # Restart SCM and will see following warnings in SCM log > 17/05/02 00:47:08 WARN node.SCMNodeManager: SCM receive heartbeat from > unregistered datanode > Datanode could not re-establish communication with SCM afterwards. Propose to > fix this by adding a new command in HB handling telling datanode to > re-register with SCM. Datanode once received this command transits to > REGISTER state again to proceed. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11830) Ozone: Datanode needs to re-register to SCM if SCM is restarted
[ https://issues.apache.org/jira/browse/HDFS-11830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017109#comment-16017109 ] Mukul Kumar Singh commented on HDFS-11830: -- +1, Looks good to me. > Ozone: Datanode needs to re-register to SCM if SCM is restarted > --- > > Key: HDFS-11830 > URL: https://issues.apache.org/jira/browse/HDFS-11830 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Critical > Attachments: HDFS-11830-HDFS-7240.001.patch, > HDFS-11830-HDFS-7240.002.patch, HDFS-11830-HDFS-7240.003.patch, > HDFS-11830-HDFS-7240.004.patch > > > Problem description: > # Start NN, DN, SCM > # Restart SCM and will see following warnings in SCM log > 17/05/02 00:47:08 WARN node.SCMNodeManager: SCM receive heartbeat from > unregistered datanode > Datanode could not re-establish communication with SCM afterwards. Propose to > fix this by adding a new command in HB handling telling datanode to > re-register with SCM. Datanode once received this command transits to > REGISTER state again to proceed. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11830) Ozone: Datanode needs to re-register to SCM if SCM is restarted
[
https://issues.apache.org/jira/browse/HDFS-11830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017031#comment-16017031
]
Hadoop QA commented on HDFS-11830:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
27s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
59s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 32s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}104m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics |
| | hadoop.cblock.TestCBlockServerPersistence |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11830 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868881/HDFS-11830-HDFS-7240.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 0e682f7ea62f 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 002bb5f |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/19506/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19506/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19506/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-11830) Ozone: Datanode needs to re-register to SCM if SCM is restarted
[ https://issues.apache.org/jira/browse/HDFS-11830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016925#comment-16016925 ] Weiwei Yang commented on HDFS-11830: Hi [~msingh] Thanks. I will fix the checkstyle issue shortly in next patch. > Ozone: Datanode needs to re-register to SCM if SCM is restarted > --- > > Key: HDFS-11830 > URL: https://issues.apache.org/jira/browse/HDFS-11830 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Critical > Attachments: HDFS-11830-HDFS-7240.001.patch, > HDFS-11830-HDFS-7240.002.patch, HDFS-11830-HDFS-7240.003.patch > > > Problem description: > # Start NN, DN, SCM > # Restart SCM and will see following warnings in SCM log > 17/05/02 00:47:08 WARN node.SCMNodeManager: SCM receive heartbeat from > unregistered datanode > Datanode could not re-establish communication with SCM afterwards. Propose to > fix this by adding a new command in HB handling telling datanode to > re-register with SCM. Datanode once received this command transits to > REGISTER state again to proceed. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11830) Ozone: Datanode needs to re-register to SCM if SCM is restarted
[ https://issues.apache.org/jira/browse/HDFS-11830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016923#comment-16016923 ] Mukul Kumar Singh commented on HDFS-11830: -- Thanks for the latest patch [~cheersyang], apart from the check style issues, the patch looks good to me. > Ozone: Datanode needs to re-register to SCM if SCM is restarted > --- > > Key: HDFS-11830 > URL: https://issues.apache.org/jira/browse/HDFS-11830 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Critical > Attachments: HDFS-11830-HDFS-7240.001.patch, > HDFS-11830-HDFS-7240.002.patch, HDFS-11830-HDFS-7240.003.patch > > > Problem description: > # Start NN, DN, SCM > # Restart SCM and will see following warnings in SCM log > 17/05/02 00:47:08 WARN node.SCMNodeManager: SCM receive heartbeat from > unregistered datanode > Datanode could not re-establish communication with SCM afterwards. Propose to > fix this by adding a new command in HB handling telling datanode to > re-register with SCM. Datanode once received this command transits to > REGISTER state again to proceed. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11830) Ozone: Datanode needs to re-register to SCM if SCM is restarted
[
https://issues.apache.org/jira/browse/HDFS-11830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016863#comment-16016863
]
Hadoop QA commented on HDFS-11830:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
55s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 36s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}102m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11830 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868866/HDFS-11830-HDFS-7240.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 26ce9081204f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 002bb5f |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/19505/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/19505/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Commented] (HDFS-11830) Ozone: Datanode needs to re-register to SCM if SCM is restarted
[ https://issues.apache.org/jira/browse/HDFS-11830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016796#comment-16016796 ] Weiwei Yang commented on HDFS-11830: Thanks [~msingh]. I just uploaded v3 patch to address your comments. bq. Can we log this as an error ? This might help in debugging issues later. I added a debug logging for this, this is not a real error so a debug message should be enough. bq. SCMNodeManager.java:141, there is an extra space between Fixed. bq. Should ReregisterCommand should extend SCMReregisterCmdResponseProto Fixed. Thank you > Ozone: Datanode needs to re-register to SCM if SCM is restarted > --- > > Key: HDFS-11830 > URL: https://issues.apache.org/jira/browse/HDFS-11830 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Critical > Attachments: HDFS-11830-HDFS-7240.001.patch, > HDFS-11830-HDFS-7240.002.patch, HDFS-11830-HDFS-7240.003.patch > > > Problem description: > # Start NN, DN, SCM > # Restart SCM and will see following warnings in SCM log > 17/05/02 00:47:08 WARN node.SCMNodeManager: SCM receive heartbeat from > unregistered datanode > Datanode could not re-establish communication with SCM afterwards. Propose to > fix this by adding a new command in HB handling telling datanode to > re-register with SCM. Datanode once received this command transits to > REGISTER state again to proceed. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11830) Ozone: Datanode needs to re-register to SCM if SCM is restarted
[
https://issues.apache.org/jira/browse/HDFS-11830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016052#comment-16016052
]
Mukul Kumar Singh commented on HDFS-11830:
--
Thanks for the second patch [~cheersyang]
1) Sure, we can ignore the register command, Can we log this as an error ? This
might help in debugging issues later.
2) nit, SCMNodeManager.java:141, there is an extra space between {
3) Should ReregisterCommand should extend SCMReregisterCmdResponseProto ?
> Ozone: Datanode needs to re-register to SCM if SCM is restarted
> ---
>
> Key: HDFS-11830
> URL: https://issues.apache.org/jira/browse/HDFS-11830
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Attachments: HDFS-11830-HDFS-7240.001.patch,
> HDFS-11830-HDFS-7240.002.patch
>
>
> Problem description:
> # Start NN, DN, SCM
> # Restart SCM and will see following warnings in SCM log
> 17/05/02 00:47:08 WARN node.SCMNodeManager: SCM receive heartbeat from
> unregistered datanode
> Datanode could not re-establish communication with SCM afterwards. Propose to
> fix this by adding a new command in HB handling telling datanode to
> re-register with SCM. Datanode once received this command transits to
> REGISTER state again to proceed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11830) Ozone: Datanode needs to re-register to SCM if SCM is restarted
[
https://issues.apache.org/jira/browse/HDFS-11830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014949#comment-16014949
]
Hadoop QA commented on HDFS-11830:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
47s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 34s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.ozone.scm.TestContainerSQLCli |
| | hadoop.cblock.TestCBlockCLI |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11830 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868611/HDFS-11830-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 0f1e48137218 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 7dbe89d |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/19478/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19478/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Commented] (HDFS-11830) Ozone: Datanode needs to re-register to SCM if SCM is restarted
[
https://issues.apache.org/jira/browse/HDFS-11830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014841#comment-16014841
]
Weiwei Yang commented on HDFS-11830:
Hello [~msingh]
Thank you for helping to review. I have addressed most of your comments in v2
patch except one
bq. We should also raise an exception if the endpoint is in any other state
apart from HEARTBEAT.
We cannot raise an exception here because in test mode, if we set a short
heartbeat interval, 1s for example. Datanode might not be able to fully transit
to {{REGISTER}} state and it receives another response from SCM with
{{reregisterCommand}} command. I think just ignore changing the state in this
case should be fine. What do you think?
Thank you.
> Ozone: Datanode needs to re-register to SCM if SCM is restarted
> ---
>
> Key: HDFS-11830
> URL: https://issues.apache.org/jira/browse/HDFS-11830
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Attachments: HDFS-11830-HDFS-7240.001.patch,
> HDFS-11830-HDFS-7240.002.patch
>
>
> Problem description:
> # Start NN, DN, SCM
> # Restart SCM and will see following warnings in SCM log
> 17/05/02 00:47:08 WARN node.SCMNodeManager: SCM receive heartbeat from
> unregistered datanode
> Datanode could not re-establish communication with SCM afterwards. Propose to
> fix this by adding a new command in HB handling telling datanode to
> re-register with SCM. Datanode once received this command transits to
> REGISTER state again to proceed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11830) Ozone: Datanode needs to re-register to SCM if SCM is restarted
[ https://issues.apache.org/jira/browse/HDFS-11830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014386#comment-16014386 ] Mukul Kumar Singh commented on HDFS-11830: -- Thanks for the patch [~cheersyang], Following are my comments 1) HeartbeatEndpointTask.java:134 I feel that the rpc endpoint state here should only be HEARTBEAT, and we should transition the state to re-register only if the current state is HEARTBEAT. We should also raise an exception if the endpoint is in any other state apart from HEARTBEAT. 2) Also can you please rename this command to reregister, this would help in differentiating with the registered command. > Ozone: Datanode needs to re-register to SCM if SCM is restarted > --- > > Key: HDFS-11830 > URL: https://issues.apache.org/jira/browse/HDFS-11830 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Critical > Attachments: HDFS-11830-HDFS-7240.001.patch > > > Problem description: > # Start NN, DN, SCM > # Restart SCM and will see following warnings in SCM log > 17/05/02 00:47:08 WARN node.SCMNodeManager: SCM receive heartbeat from > unregistered datanode > Datanode could not re-establish communication with SCM afterwards. Propose to > fix this by adding a new command in HB handling telling datanode to > re-register with SCM. Datanode once received this command transits to > REGISTER state again to proceed. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11830) Ozone: Datanode needs to re-register to SCM if SCM is restarted
[
https://issues.apache.org/jira/browse/HDFS-11830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16013868#comment-16013868
]
Hadoop QA commented on HDFS-11830:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
22s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 45s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot |
| | org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| | org.apache.hadoop.hdfs.server.namenode.TestCheckpoint |
| | org.apache.hadoop.hdfs.server.namenode.TestFSEditLogLoader |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11830 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868493/HDFS-11830-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux ea25b66c89c6 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 7dbe89d |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/19465/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19465/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
