[jira] [Commented] (HDDS-737) Introduce Incremental Container Report
[ https://issues.apache.org/jira/browse/HDDS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679746#comment-16679746 ] Hudson commented on HDDS-737: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15387 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15387/]) HDDS-737. Introduce Incremental Container Report. Contributed by Nanda (nanda: rev c80f753b0e95eb722a972f836c1e4d16fd823434) * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerManager.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestPipelineClose.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/command/TestCommandStatusReportHandler.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/events/SCMEvents.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/StaleNodeHandler.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/ozone/container/testutils/ReplicationNodeManagerMock.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDatanodeHeartbeatDispatcher.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDatanodeProtocolServer.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeStateManager.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerReportHandler.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestBlockDeletion.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/MockNodeManager.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestReplicationManager.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/states/NodeStateMap.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestNode2PipelineMap.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeStateMachine.java * (add) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/IncrementalContainerReportHandler.java * (delete) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/ozone/container/TestCloseContainerWatcher.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/ozone/container/common/TestEndPoint.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/HeartbeatEndpointTask.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java * (edit) hadoop-hdds/container-service/src/main/proto/StorageContainerDatanodeProtocol.proto * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NewNodeHandler.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerStateManager.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/CloseContainerCommandHandler.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestDeadNodeHandler.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/states/ContainerStateMap.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/SCMContainerManager.java * (delete) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/CloseContainerWatcher.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerInfo.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeReportHandler.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/CloseContainerEventHandler.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerReportHandler.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSCMContainerManager.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerReplica.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DeadNodeHandler.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/command/CommandStatusReportHandler.java * (edit)
[jira] [Commented] (HDDS-737) Introduce Incremental Container Report
[ https://issues.apache.org/jira/browse/HDDS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679732#comment-16679732 ] Nanda kumar commented on HDDS-737: -- Tested it locally, failures are not related to this patch. I will commit it shortly. > Introduce Incremental Container Report > -- > > Key: HDDS-737 > URL: https://issues.apache.org/jira/browse/HDDS-737 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode, SCM >Reporter: Nanda kumar >Assignee: Nanda kumar >Priority: Major > Attachments: HDDS-737.000.patch, HDDS-737.001.patch > > > We will use Incremental Container Report (ICR) to immediately inform SCM when > there is some state change to the container in datanode. This will make sure > that SCM is updated as soon as the state of a container changes and doesn’t > have to wait for full container report. > *When do we send ICR?* > * When a container replica state changes from open/closing to closed > * When a container replica state changes from open/closing to quasi closed > * When a container replica state changes from quasi closed to closed > * When a container replica is deleted in datanode > * When a container replica is copied from another datanode > * When a container replica is discovered to be corrupted -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-737) Introduce Incremental Container Report
[ https://issues.apache.org/jira/browse/HDDS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679368#comment-16679368 ] Jitendra Nath Pandey commented on HDDS-737: --- I am +1 for the patch, if failures are not related to the patch. > Introduce Incremental Container Report > -- > > Key: HDDS-737 > URL: https://issues.apache.org/jira/browse/HDDS-737 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode, SCM >Reporter: Nanda kumar >Assignee: Nanda kumar >Priority: Major > Attachments: HDDS-737.000.patch, HDDS-737.001.patch > > > We will use Incremental Container Report (ICR) to immediately inform SCM when > there is some state change to the container in datanode. This will make sure > that SCM is updated as soon as the state of a container changes and doesn’t > have to wait for full container report. > *When do we send ICR?* > * When a container replica state changes from open/closing to closed > * When a container replica state changes from open/closing to quasi closed > * When a container replica state changes from quasi closed to closed > * When a container replica is deleted in datanode > * When a container replica is copied from another datanode > * When a container replica is discovered to be corrupted -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-737) Introduce Incremental Container Report
[ https://issues.apache.org/jira/browse/HDDS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679739#comment-16679739 ] Nanda kumar commented on HDDS-737: -- Thanks [~linyiqun] & [~jnp] for the reviews. Committed it to trunk. > Introduce Incremental Container Report > -- > > Key: HDDS-737 > URL: https://issues.apache.org/jira/browse/HDDS-737 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode, SCM >Reporter: Nanda kumar >Assignee: Nanda kumar >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-737.000.patch, HDDS-737.001.patch > > > We will use Incremental Container Report (ICR) to immediately inform SCM when > there is some state change to the container in datanode. This will make sure > that SCM is updated as soon as the state of a container changes and doesn’t > have to wait for full container report. > *When do we send ICR?* > * When a container replica state changes from open/closing to closed > * When a container replica state changes from open/closing to quasi closed > * When a container replica state changes from quasi closed to closed > * When a container replica is deleted in datanode > * When a container replica is copied from another datanode > * When a container replica is discovered to be corrupted -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-737) Introduce Incremental Container Report
[
https://issues.apache.org/jira/browse/HDDS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677790#comment-16677790
]
Nanda kumar commented on HDDS-737:
--
The shadedclient build is failing even without the patch in Jenkins run
https://builds.apache.org/job/PreCommit-HDDS-Build/1629/artifact/out/branch-shadedclient.txt
I thinks it affect the unit test run, whenever shadedclient build is failing
none of the unit test are running and the build is failing with
{code}
[ERROR] ExecutionException The forked VM terminated without properly saying
goodbye. VM crash or System.exit called?
{code}
> Introduce Incremental Container Report
> --
>
> Key: HDDS-737
> URL: https://issues.apache.org/jira/browse/HDDS-737
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode, SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-737.000.patch, HDDS-737.001.patch
>
>
> We will use Incremental Container Report (ICR) to immediately inform SCM when
> there is some state change to the container in datanode. This will make sure
> that SCM is updated as soon as the state of a container changes and doesn’t
> have to wait for full container report.
> *When do we send ICR?*
> * When a container replica state changes from open/closing to closed
> * When a container replica state changes from open/closing to quasi closed
> * When a container replica state changes from quasi closed to closed
> * When a container replica is deleted in datanode
> * When a container replica is copied from another datanode
> * When a container replica is discovered to be corrupted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-737) Introduce Incremental Container Report
[
https://issues.apache.org/jira/browse/HDDS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677654#comment-16677654
]
Hadoop QA commented on HDDS-737:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 14 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
49s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 17m
53s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 13s{color} | {color:orange} root: The patch generated 1 new + 9 unchanged -
1 fixed = 10 total (was 10) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 11m
16s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 44s{color}
| {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 42s{color}
| {color:red} container-service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 39s{color}
| {color:red} server-scm in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 41s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}110m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce
[jira] [Commented] (HDDS-737) Introduce Incremental Container Report
[
https://issues.apache.org/jira/browse/HDDS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677529#comment-16677529
]
Hadoop QA commented on HDDS-737:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 14 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 7m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
23s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 16m
2s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 3s{color} | {color:orange} root: The patch generated 1 new + 9 unchanged -
1 fixed = 10 total (was 10) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 9m
48s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 35s{color}
| {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 34s{color}
| {color:red} container-service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 33s{color}
| {color:red} server-scm in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 31s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}109m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce
[jira] [Commented] (HDDS-737) Introduce Incremental Container Report
[
https://issues.apache.org/jira/browse/HDDS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677270#comment-16677270
]
Hadoop QA commented on HDDS-737:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 14 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
37s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 17m
36s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 15s{color} | {color:orange} root: The patch generated 1 new + 9 unchanged -
1 fixed = 10 total (was 10) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 11m
16s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 45s{color}
| {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 42s{color}
| {color:red} container-service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 40s{color}
| {color:red} server-scm in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 40s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}109m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce
[jira] [Commented] (HDDS-737) Introduce Incremental Container Report
[
https://issues.apache.org/jira/browse/HDDS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677143#comment-16677143
]
Nanda kumar commented on HDDS-737:
--
Looks like Jenkins run has some problem, the build failed with the below error
{code:java}
[ERROR] Please refer to
/testptch/hadoop/hadoop-hdds/common/target/surefire-reports for the individual
test results.
[ERROR] Please refer to dump files (if any exist) [date]-jvmRun[N].dump,
[date].dumpstream and [date]-jvmRun[N].dumpstream.
[ERROR] ExecutionException The forked VM terminated without properly saying
goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /testptch/hadoop/hadoop-hdds/common &&
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx2048m
-XX:+HeapDumpOnOutOfMemoryError -DminiClusterDedicatedDirs=true -jar
/testptch/hadoop/hadoop-hdds/common/target/surefire/surefirebooter1364262303668308594.jar
/testptch/hadoop/hadoop-hdds/common/target/surefire
2018-11-06T11-45-30_771-jvmRun4 surefire2576372466595873598tmp
surefire_22763712882772709533tmp
{code}
Re-triggered Jenkins pre-commit build:
[https://builds.apache.org/job/PreCommit-HDDS-Build/1626/]
> Introduce Incremental Container Report
> --
>
> Key: HDDS-737
> URL: https://issues.apache.org/jira/browse/HDDS-737
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode, SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-737.000.patch, HDDS-737.001.patch
>
>
> We will use Incremental Container Report (ICR) to immediately inform SCM when
> there is some state change to the container in datanode. This will make sure
> that SCM is updated as soon as the state of a container changes and doesn’t
> have to wait for full container report.
> *When do we send ICR?*
> * When a container replica state changes from open/closing to closed
> * When a container replica state changes from open/closing to quasi closed
> * When a container replica state changes from quasi closed to closed
> * When a container replica is deleted in datanode
> * When a container replica is copied from another datanode
> * When a container replica is discovered to be corrupted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-737) Introduce Incremental Container Report
[
https://issues.apache.org/jira/browse/HDDS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676669#comment-16676669
]
Hadoop QA commented on HDDS-737:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 14 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 7m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
52s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 17m
47s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 17s{color} | {color:orange} root: The patch generated 1 new + 9 unchanged -
1 fixed = 10 total (was 10) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 11m
10s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 46s{color}
| {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 42s{color}
| {color:red} container-service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 40s{color}
| {color:red} server-scm in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 40s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce
[jira] [Commented] (HDDS-737) Introduce Incremental Container Report
[
https://issues.apache.org/jira/browse/HDDS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676668#comment-16676668
]
Hadoop QA commented on HDDS-737:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 14 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 7m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
53s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 18m
40s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
27s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 16s{color} | {color:orange} root: The patch generated 1 new + 9 unchanged -
1 fixed = 10 total (was 10) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 10m
38s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 38s{color}
| {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 33s{color}
| {color:red} container-service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 33s{color}
| {color:red} server-scm in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 32s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}115m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce
[jira] [Commented] (HDDS-737) Introduce Incremental Container Report
[
https://issues.apache.org/jira/browse/HDDS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676432#comment-16676432
]
Nanda kumar commented on HDDS-737:
--
[~jnp], thanks for the review.
{quote}In CloseContainerCommandHandler#handle the container state should be set
to CLOSING before making a ratis call.
{quote}
This is done as part of HDDS-801.
{quote}pipelineManager is set in ContainerReportHandler but never used.
{quote}
Both in ContainerReportHandler and IncrementalContainerReportHandler,
pipelineManager will be required when we handle state change. We need to remove
the container from OPEN pipeline when the container is moved to CLOSED state.
For now, added TODO in both the classes. When we handle state change,
pipelineManager will be used.
{quote}Heartbeating thread can also receive interrupt when shutting down
{quote}
Good catch. Updated the comment.
{quote}NewNodeHandler does nothing. Shouldn't it send command for a container
report?
{quote}
NewNode event is triggered by NodeManager, it has already made an entry for the
registered node in NodeStateManager. We get container report as part of
register call, and that container report will be processed by
ContainerReportHandler to update the container replica state. We currently have
nothing to do when we receive a new node event from NodeManager. NewNodeHandler
is just a placeholder for now, in future, if required, we can use it.
{quote}Why is removeNode removed from NodeManager? It seems like the right
place.
{quote}
We currently don't remove a node from NodeManager once it is registered. We can
add removeNode logic when we implement decommissioning of a datanode. (existing
removeNode logic was incomplete).
[~linyiqun]
{quote}I prefer to add additional try-catch for thread.sleep and get
InterruptedException.
{quote}
Since we also have to handle {{InterruptedException}} when the shutdown is
initiated, I feel it is better to have try-catch for the complete code inside
the while loop.
> Introduce Incremental Container Report
> --
>
> Key: HDDS-737
> URL: https://issues.apache.org/jira/browse/HDDS-737
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode, SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-737.000.patch, HDDS-737.001.patch
>
>
> We will use Incremental Container Report (ICR) to immediately inform SCM when
> there is some state change to the container in datanode. This will make sure
> that SCM is updated as soon as the state of a container changes and doesn’t
> have to wait for full container report.
> *When do we send ICR?*
> * When a container replica state changes from open/closing to closed
> * When a container replica state changes from open/closing to quasi closed
> * When a container replica state changes from quasi closed to closed
> * When a container replica is deleted in datanode
> * When a container replica is copied from another datanode
> * When a container replica is discovered to be corrupted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-737) Introduce Incremental Container Report
[
https://issues.apache.org/jira/browse/HDDS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676138#comment-16676138
]
Jitendra Nath Pandey commented on HDDS-737:
---
Overall the patch looks pretty good, a few comments.
# In {{CloseContainerCommandHandler#handle}} the container state should be set
to CLOSING before making a ratis call.
# {{pipelineManager}} is set in ContainerReportHandler but never used.
# Heartbeating thread can also receive interrupt when shutting down, the
comment added in {{DatanodeStateMachine#start}} is not accurate.
# {{NewNodeHandler}} does nothing. Shouldn't it send command for a container
report?
# Why is {{removeNode}} removed from NodeManager? It seems like the right
place.
> Introduce Incremental Container Report
> --
>
> Key: HDDS-737
> URL: https://issues.apache.org/jira/browse/HDDS-737
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode, SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-737.000.patch
>
>
> We will use Incremental Container Report (ICR) to immediately inform SCM when
> there is some state change to the container in datanode. This will make sure
> that SCM is updated as soon as the state of a container changes and doesn’t
> have to wait for full container report.
> *When do we send ICR?*
> * When a container replica state changes from open/closing to closed
> * When a container replica state changes from open/closing to quasi closed
> * When a container replica state changes from quasi closed to closed
> * When a container replica is deleted in datanode
> * When a container replica is copied from another datanode
> * When a container replica is discovered to be corrupted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-737) Introduce Incremental Container Report
[
https://issues.apache.org/jira/browse/HDDS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675615#comment-16675615
]
Nanda kumar commented on HDDS-737:
--
[~linyiqun], thanks for the review.
bq. I prefer to add additional try-catch for thread.sleep and get
InterruptedException.
Good idea. Will address it in the next patch.
bq. Can we have a new unit test for the incremental container report?
I have added {{TestCloseContainerCommandHandler}} in HDDS-801 which tests
whether ICR is properly triggered.
HDDS-801 also introduces {{OzoneContainer#updateContainerState}} call which
triggers ICR, there is a bit of restructuring of code in the way we trigger ICR
in HDDS-801.
> Introduce Incremental Container Report
> --
>
> Key: HDDS-737
> URL: https://issues.apache.org/jira/browse/HDDS-737
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode, SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-737.000.patch
>
>
> We will use Incremental Container Report (ICR) to immediately inform SCM when
> there is some state change to the container in datanode. This will make sure
> that SCM is updated as soon as the state of a container changes and doesn’t
> have to wait for full container report.
> *When do we send ICR?*
> * When a container replica state changes from open/closing to closed
> * When a container replica state changes from open/closing to quasi closed
> * When a container replica state changes from quasi closed to closed
> * When a container replica is deleted in datanode
> * When a container replica is copied from another datanode
> * When a container replica is discovered to be corrupted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-737) Introduce Incremental Container Report
[
https://issues.apache.org/jira/browse/HDDS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674905#comment-16674905
]
Yiqun Lin commented on HDDS-737:
Thanks for giving the summary note, [~nandakumar131]. It helps me a lot.
I have one comments for the behaviour for ICR triggering. I prefer to add
additional try-catch for {{thread.sleep}} and get InterruptedException. At
least, we can know there is a triggering for the ICR.
Can we have a new unit test for the incremental container report? Major change
of the test is the refactoring. More test cases we can complete gradually in
follow-up JIARs
> Introduce Incremental Container Report
> --
>
> Key: HDDS-737
> URL: https://issues.apache.org/jira/browse/HDDS-737
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode, SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-737.000.patch
>
>
> We will use Incremental Container Report (ICR) to immediately inform SCM when
> there is some state change to the container in datanode. This will make sure
> that SCM is updated as soon as the state of a container changes and doesn’t
> have to wait for full container report.
> *When do we send ICR?*
> * When a container replica state changes from open/closing to closed
> * When a container replica state changes from open/closing to quasi closed
> * When a container replica state changes from quasi closed to closed
> * When a container replica is deleted in datanode
> * When a container replica is copied from another datanode
> * When a container replica is discovered to be corrupted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-737) Introduce Incremental Container Report
[
https://issues.apache.org/jira/browse/HDDS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674422#comment-16674422
]
Nanda kumar commented on HDDS-737:
--
Thanks [~linyiqun] for taking a look at the patch.
{quote}could you give a summary note of the change
{quote}
Sure.
* The main change in this patch is to introduce a way to send ICR immediately.
This is done by introducing {{triggerHeartbeat}} method in
{{DatanodeStateMachine}}, this will immediately trigger a heartbeat to SCM
which will also include the reports which are ready to send. So, to send an ICR
immediately what we have to do is
1) Add ICR (Container Report) to {{StateContext}}.
2) Call {{triggerHeartbeat}} method.
* Since we have ICR in place, we don't need to send command status for
{{CloseContainerCommand}}. (We eventually want to remove the command status for
all the commands)
This patch removes the command status logic for {{CloseContainerCommand}} and
also removes the command watcher (CloseContainerWatcher) for
{{CloseContainerCommand}} in SCM.
* Added IncrementalContainerReportHandler in SCM. (It is not complete yet,
added TODO. Needs follow up jiras)
* Processing of container report was previously done by {{NodeManager}} and
{{ContainerManager}}. This logic is moved to {{ContainerReportHandler}} (There
is a TODO which needs follow up jira)
* Few more refactorings like removing Node2Container class and moving that
data structure to {{NodeStateMap}}.
I wanted to cover all the scenarios in this jira itself, but the patch already
got huge and it will become very difficult to review. I will start filing
follow up jiras.
{quote}Current change has addressed all the points (When do we send ICR)
mentioned in JIRA's description?
{quote}
Only one scenario is handled in this jira
* When a container replica state changes from open/closing to closed
We still don't have code to QUASI_CLOSE a container in datanode, we should
trigger ICR when we do that. (HDDS-801)
The same is for deleted, copied or corrupted.
I will start filing jiras so that we can keep track of it.
Thanks a lot for spending your time reviewing the patch.
> Introduce Incremental Container Report
> --
>
> Key: HDDS-737
> URL: https://issues.apache.org/jira/browse/HDDS-737
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode, SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-737.000.patch
>
>
> We will use Incremental Container Report (ICR) to immediately inform SCM when
> there is some state change to the container in datanode. This will make sure
> that SCM is updated as soon as the state of a container changes and doesn’t
> have to wait for full container report.
> *When do we send ICR?*
> * When a container replica state changes from open/closing to closed
> * When a container replica state changes from open/closing to quasi closed
> * When a container replica state changes from quasi closed to closed
> * When a container replica is deleted in datanode
> * When a container replica is copied from another datanode
> * When a container replica is discovered to be corrupted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-737) Introduce Incremental Container Report
[ https://issues.apache.org/jira/browse/HDDS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674001#comment-16674001 ] Yiqun Lin commented on HDDS-737: Haven't fully reviewed for this, looks like the patch is a little large, :). [~nandakumar131], could you give a summary note of the change in your patch? It will be very helpful for others to review. >From a quick glance for the patch.there are lots of deletion logic. Current >change has addressed all the points (*When do we send ICR*) mentioned in >JIRA's description? > Introduce Incremental Container Report > -- > > Key: HDDS-737 > URL: https://issues.apache.org/jira/browse/HDDS-737 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode, SCM >Reporter: Nanda kumar >Assignee: Nanda kumar >Priority: Major > Attachments: HDDS-737.000.patch > > > We will use Incremental Container Report (ICR) to immediately inform SCM when > there is some state change to the container in datanode. This will make sure > that SCM is updated as soon as the state of a container changes and doesn’t > have to wait for full container report. > *When do we send ICR?* > * When a container replica state changes from open/closing to closed > * When a container replica state changes from open/closing to quasi closed > * When a container replica state changes from quasi closed to closed > * When a container replica is deleted in datanode > * When a container replica is copied from another datanode > * When a container replica is discovered to be corrupted -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-737) Introduce Incremental Container Report
[
https://issues.apache.org/jira/browse/HDDS-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673685#comment-16673685
]
Hadoop QA commented on HDDS-737:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 14 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
43s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 17m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
6s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 45s{color} | {color:orange} root: The patch generated 6 new + 9 unchanged -
1 fixed = 15 total (was 10) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
8s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
57s{color} | {color:green} container-service in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 30s{color}
| {color:red} server-scm in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
16s{color} | {color:green} integration-test in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}128m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
