[jira] [Commented] (HDDS-1025) Handle replication of closed containers in DeadNodeHanlder
[ https://issues.apache.org/jira/browse/HDDS-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757939#comment-16757939 ] Hudson commented on HDDS-1025: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15864 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15864/]) HDDS-1025. Handle replication of closed containers in DeadNodeHanlder. (yqlin: rev 16195eaee1b4a7620bc018f48e9c24fc5fc7cc02) * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DeadNodeHandler.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/TestUtils.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestDeadNodeHandler.java > Handle replication of closed containers in DeadNodeHanlder > -- > > Key: HDDS-1025 > URL: https://issues.apache.org/jira/browse/HDDS-1025 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Attachments: HDDS-1025.00.patch, HDDS-1025.01.patch, > HDDS-1025.02.patch > > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to do one of the TODO mentioned in the DeadNodeHandler > // TODO: Check replica count and call replication manager. > > As Right now, when a node is dead, replication for the closed containers is > not triggered. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1025) Handle replication of closed containers in DeadNodeHanlder
[ https://issues.apache.org/jira/browse/HDDS-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757927#comment-16757927 ] Yiqun Lin commented on HDDS-1025: - LGTM, +1. Thanks for addressing the comment, [~bharatviswa]! > Handle replication of closed containers in DeadNodeHanlder > -- > > Key: HDDS-1025 > URL: https://issues.apache.org/jira/browse/HDDS-1025 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Attachments: HDDS-1025.00.patch, HDDS-1025.01.patch, > HDDS-1025.02.patch > > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to do one of the TODO mentioned in the DeadNodeHandler > // TODO: Check replica count and call replication manager. > > As Right now, when a node is dead, replication for the closed containers is > not triggered. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1025) Handle replication of closed containers in DeadNodeHanlder
[
https://issues.apache.org/jira/browse/HDDS-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757723#comment-16757723
]
Hadoop QA commented on HDDS-1025:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 8m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 37m
35s{color} | {color:green} hadoop-ozone in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
30s{color} | {color:green} hadoop-hdds in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1025 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12957166/HDDS-1025.02.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle |
| uname | Linux 35e32bf88e49 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh |
| git revision | trunk / f738b39 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/2158/testReport/ |
| Max. process+thread count | 1129 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/2158/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Handle replication of closed containers in DeadNodeHanlder
> --
>
> Key: HDDS-1025
> URL: https://issues.apache.org/jira/browse/HDDS-1025
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Attachments: HDDS-1025.00.patch, HDDS-1025.01.patch,
> HDDS-1025.02.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This Jira is to do one of the TODO mentioned in the DeadNodeHandler
> // TODO: Check replica count and call replication manager.
>
> As Right now, when a node is dead, replication for the closed containers is
> not
[jira] [Commented] (HDDS-1025) Handle replication of closed containers in DeadNodeHanlder
[
https://issues.apache.org/jira/browse/HDDS-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757714#comment-16757714
]
Hadoop QA commented on HDDS-1025:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 30m 52s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
33s{color} | {color:green} hadoop-hdds in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1025 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12957165/HDDS-1025.01.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle |
| uname | Linux 49ae10342f9e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh |
| git revision | trunk / f738b39 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/2157/artifact/out/patch-unit-hadoop-ozone.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/2157/testReport/ |
| Max. process+thread count | 1136 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/2157/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Handle replication of closed containers in DeadNodeHanlder
> --
>
> Key: HDDS-1025
> URL: https://issues.apache.org/jira/browse/HDDS-1025
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Attachments: HDDS-1025.00.patch, HDDS-1025.01.patch,
> HDDS-1025.02.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This Jira is to
[jira] [Commented] (HDDS-1025) Handle replication of closed containers in DeadNodeHanlder
[ https://issues.apache.org/jira/browse/HDDS-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757684#comment-16757684 ] Bharat Viswanadham commented on HDDS-1025: -- Thank you [~linyiqun] for review. Addressed the review comments in patch v02 Not added LOG for container open, as we plan to open the Jira and fix the TODO, but checked for open containers we don't have the request for open container in the log. > Handle replication of closed containers in DeadNodeHanlder > -- > > Key: HDDS-1025 > URL: https://issues.apache.org/jira/browse/HDDS-1025 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Attachments: HDDS-1025.00.patch, HDDS-1025.01.patch, > HDDS-1025.02.patch > > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to do one of the TODO mentioned in the DeadNodeHandler > // TODO: Check replica count and call replication manager. > > As Right now, when a node is dead, replication for the closed containers is > not triggered. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1025) Handle replication of closed containers in DeadNodeHanlder
[
https://issues.apache.org/jira/browse/HDDS-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757043#comment-16757043
]
Yiqun Lin commented on HDDS-1025:
-
The patch almost looks good to me, I afraid that there is a chance to lead the
log flooding. I prefer to use debug/trace log to print this.
{noformat}
LOG.debug("Replicate Request fired for container {}, exisiting " +
+"replica count {}, expected replica count {}",
+container.getContainerID(), existingReplicas, expectedReplicas);
{noformat}
In additional, for the open containers if we find, we can print a debug log as
well. This message info can be used to verif in UT.
{code:java}
// TODO: For open containers, trigger close on other nodes
// TODO: Check replica count and call replication manager
// on these containers.
if (!container.isOpen()) {
//..
} else {
LOG.debug(""); <=== print the log
}
{code}
In the test, we can add {{GenericTestUtils.setLogLevel(DeadNodeHandler.LOG,
Level.DEBUG);}} to enable the log printing.
> Handle replication of closed containers in DeadNodeHanlder
> --
>
> Key: HDDS-1025
> URL: https://issues.apache.org/jira/browse/HDDS-1025
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Attachments: HDDS-1025.00.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This Jira is to do one of the TODO mentioned in the DeadNodeHandler
> // TODO: Check replica count and call replication manager.
>
> As Right now, when a node is dead, replication for the closed containers is
> not triggered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1025) Handle replication of closed containers in DeadNodeHanlder
[
https://issues.apache.org/jira/browse/HDDS-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756728#comment-16756728
]
Hadoop QA commented on HDDS-1025:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 30m 51s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
41s{color} | {color:green} hadoop-hdds in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis |
| | hadoop.ozone.container.TestContainerReplication |
| | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1025 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12956971/HDDS-1025.00.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle |
| uname | Linux 7a4359594b8a 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh |
| git revision | trunk / c354195 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/2149/artifact/out/patch-unit-hadoop-ozone.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/2149/testReport/ |
| Max. process+thread count | 1133 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/2149/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Handle replication of closed containers in DeadNodeHanlder
> --
>
> Key: HDDS-1025
> URL: https://issues.apache.org/jira/browse/HDDS-1025
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDDS-1025.00.patch
>
>
> This Jira is to do one of the TODO mentioned
[jira] [Commented] (HDDS-1025) Handle replication of closed containers in DeadNodeHanlder
[ https://issues.apache.org/jira/browse/HDDS-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756681#comment-16756681 ] Bharat Viswanadham commented on HDDS-1025: -- This patch has taken care of only closed/quasi_close containers. If their expected replica count is not matching with expected, firing a ReplicaRequest event. > Handle replication of closed containers in DeadNodeHanlder > -- > > Key: HDDS-1025 > URL: https://issues.apache.org/jira/browse/HDDS-1025 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDDS-1025.00.patch > > > This Jira is to do one of the TODO mentioned in the DeadNodeHandler > // TODO: Check replica count and call replication manager. > > As Right now, when a node is dead, replication for the closed containers is > not triggered. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
