[jira] [Commented] (HDFS-12474) Ozone: SCM: Handling container report with key count and container usage.
[ https://issues.apache.org/jira/browse/HDFS-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455316#comment-16455316 ] Hudson commented on HDFS-12474: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14070 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14070/]) HDFS-12474. Ozone: SCM: Handling container report with key count and (omalley: rev 709d56fdc9ef1a5ca2c0a8f7461ee209c07dd2e1) * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/container/common/helpers/ContainerInfo.java * (delete) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/container/common/helpers/BlockContainerInfo.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/node/SCMNodeManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/ozone-default.xml * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/container/ContainerMapping.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/protocol/StorageContainerNodeProtocol.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/StorageContainerManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/Ozone.proto * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/block/BlockManagerImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/ScmConfigKeys.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/scm/container/TestContainerStateManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/container/ContainerStateManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/scm/container/TestContainerMapping.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/container/Mapping.java > Ozone: SCM: Handling container report with key count and container usage. > - > > Key: HDFS-12474 > URL: https://issues.apache.org/jira/browse/HDFS-12474 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Nanda kumar >Priority: Major > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12474-HDFS-7240.000.patch, > HDFS-12474-HDFS-7240.001.patch, HDFS-12474-HDFS-7240.002.patch, > HDFS-12474-HDFS-7240.003.patch > > > Currently, the container report only contains the # of reports sent to SCM. > We will need to provide the key count and the usage of each individual > containers to update the SCM container state maintained by > ContainerStateManager. This has a dependency on HDFS-12387. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12474) Ozone: SCM: Handling container report with key count and container usage.
[ https://issues.apache.org/jira/browse/HDFS-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451104#comment-16451104 ] Hudson commented on HDFS-12474: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14057 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14057/]) HDFS-12474. Ozone: SCM: Handling container report with key count and (xyao: rev 8107e52db5a62804712b676f9e3e14187bba9096) * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/container/common/helpers/ContainerInfo.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/container/ContainerMapping.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/StorageContainerManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/ScmConfigKeys.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/block/BlockManagerImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/scm/container/TestContainerStateManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/protocol/StorageContainerNodeProtocol.java * (delete) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/container/common/helpers/BlockContainerInfo.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/container/ContainerStateManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/container/Mapping.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/node/SCMNodeManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/ozone-default.xml * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/scm/container/TestContainerMapping.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/Ozone.proto > Ozone: SCM: Handling container report with key count and container usage. > - > > Key: HDFS-12474 > URL: https://issues.apache.org/jira/browse/HDFS-12474 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Nanda kumar >Priority: Major > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12474-HDFS-7240.000.patch, > HDFS-12474-HDFS-7240.001.patch, HDFS-12474-HDFS-7240.002.patch, > HDFS-12474-HDFS-7240.003.patch > > > Currently, the container report only contains the # of reports sent to SCM. > We will need to provide the key count and the usage of each individual > containers to update the SCM container state maintained by > ContainerStateManager. This has a dependency on HDFS-12387. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12474) Ozone: SCM: Handling container report with key count and container usage.
[
https://issues.apache.org/jira/browse/HDFS-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241005#comment-16241005
]
Hadoop QA commented on HDFS-12474:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
10s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
12s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 45s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
6 unchanged - 0 fixed = 8 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
43s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 38m 31s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
25s{color} | {color:red} The patch generated 119 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Unreaped Processes | hadoop-hdfs:6 |
| Failed junit tests | hadoop.ozone.TestMiniOzoneCluster |
| | hadoop.ozone.scm.TestSCMMXBean |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.ozone.scm.TestXceiverClientMetrics |
| | hadoop.ozone.tools.TestCorona |
| | hadoop.ozone.scm.TestSCMCli |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.ozone.TestStorageContainerManager |
| | hadoop.ozone.scm.TestAllocateContainer |
| Timed out junit tests |
[jira] [Commented] (HDFS-12474) Ozone: SCM: Handling container report with key count and container usage.
[ https://issues.apache.org/jira/browse/HDFS-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240824#comment-16240824 ] Nanda kumar commented on HDFS-12474: Thanks for the update [~xyao]. Uploaded patch v003 after rebase. > Ozone: SCM: Handling container report with key count and container usage. > - > > Key: HDFS-12474 > URL: https://issues.apache.org/jira/browse/HDFS-12474 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Nanda kumar > Labels: ozoneMerge > Attachments: HDFS-12474-HDFS-7240.000.patch, > HDFS-12474-HDFS-7240.001.patch, HDFS-12474-HDFS-7240.002.patch, > HDFS-12474-HDFS-7240.003.patch > > > Currently, the container report only contains the # of reports sent to SCM. > We will need to provide the key count and the usage of each individual > containers to update the SCM container state maintained by > ContainerStateManager. This has a dependency on HDFS-12387. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12474) Ozone: SCM: Handling container report with key count and container usage.
[ https://issues.apache.org/jira/browse/HDFS-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238597#comment-16238597 ] Xiaoyu Yao commented on HDFS-12474: --- [~nandakumar131], can you rebase the patch as it does not apply anymore? Thanks! > Ozone: SCM: Handling container report with key count and container usage. > - > > Key: HDFS-12474 > URL: https://issues.apache.org/jira/browse/HDFS-12474 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Nanda kumar >Priority: Major > Labels: ozoneMerge > Attachments: HDFS-12474-HDFS-7240.000.patch, > HDFS-12474-HDFS-7240.001.patch, HDFS-12474-HDFS-7240.002.patch > > > Currently, the container report only contains the # of reports sent to SCM. > We will need to provide the key count and the usage of each individual > containers to update the SCM container state maintained by > ContainerStateManager. This has a dependency on HDFS-12387. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12474) Ozone: SCM: Handling container report with key count and container usage.
[
https://issues.apache.org/jira/browse/HDFS-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236141#comment-16236141
]
Hadoop QA commented on HDFS-12474:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
0s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
1s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
9s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
6s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 51s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
6 unchanged - 0 fixed = 7 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
45s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 34m 9s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
23s{color} | {color:red} The patch generated 142 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 |
| | hadoop.hdfs.TestErasureCodingPolicies |
| | hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes |
| | hadoop.hdfs.TestMaintenanceState |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy |
| | hadoop.hdfs.web.TestWebHDFS |
| | hadoop.hdfs.web.TestWebHdfsWithRestCsrfPreventionFilter |
| | hadoop.hdfs.TestHDFSFileSystemContract |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 |
| |
[jira] [Commented] (HDFS-12474) Ozone: SCM: Handling container report with key count and container usage.
[
https://issues.apache.org/jira/browse/HDFS-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235231#comment-16235231
]
Hadoop QA commented on HDFS-12474:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
40s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
58s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 44s{color} | {color:orange} hadoop-hdfs-project: The patch generated 19 new
+ 6 unchanged - 0 fixed = 25 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 58s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
24s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 26s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.TestDistributedFileSystemWithECFileWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.ozone.scm.TestSCMCli |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| |
[jira] [Commented] (HDFS-12474) Ozone: SCM: Handling container report with key count and container usage.
[ https://issues.apache.org/jira/browse/HDFS-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16234876#comment-16234876 ] Xiaoyu Yao commented on HDFS-12474: --- Thanks [~nandakumar131] for the update. +1 for the v001 patch. I will commit it tomorrow if [~linyiqun] and others don't have additional comments. > Ozone: SCM: Handling container report with key count and container usage. > - > > Key: HDFS-12474 > URL: https://issues.apache.org/jira/browse/HDFS-12474 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Nanda kumar >Priority: Major > Labels: ozoneMerge > Attachments: HDFS-12474-HDFS-7240.000.patch, > HDFS-12474-HDFS-7240.001.patch > > > Currently, the container report only contains the # of reports sent to SCM. > We will need to provide the key count and the usage of each individual > containers to update the SCM container state maintained by > ContainerStateManager. This has a dependency on HDFS-12387. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12474) Ozone: SCM: Handling container report with key count and container usage.
[ https://issues.apache.org/jira/browse/HDFS-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16227716#comment-16227716 ] Nanda kumar commented on HDFS-12474: Follow up jiras * HDFS-12751 - Ozone: SCM: update container allocated size to container db for all the open containers in ContainerStateManager#close * HDFS-12752 - Ozone: SCM: Make container report processing asynchronous > Ozone: SCM: Handling container report with key count and container usage. > - > > Key: HDFS-12474 > URL: https://issues.apache.org/jira/browse/HDFS-12474 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Nanda kumar > Labels: ozoneMerge > Attachments: HDFS-12474-HDFS-7240.000.patch, > HDFS-12474-HDFS-7240.001.patch > > > Currently, the container report only contains the # of reports sent to SCM. > We will need to provide the key count and the usage of each individual > containers to update the SCM container state maintained by > ContainerStateManager. This has a dependency on HDFS-12387. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12474) Ozone: SCM: Handling container report with key count and container usage.
[
https://issues.apache.org/jira/browse/HDFS-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16227704#comment-16227704
]
Nanda kumar commented on HDFS-12474:
Thanks [~linyiqun] & [~xyao] for the review. Uploaded patch v001 addressing the
review comments.
>> line 230: Why not use {{float}} type value for the threshold-percentage
>> setting?
Changed.
>> line 375: {{(containerInfo.getUsed() / containerInfo.getSize())}} should
>> be...
Fixed.
>> line 369: ContainerInfo.lastUsed doesn't be set here.
We are setting lastUsed in {{ContainerInfo#Constructor}}, line 58
{{this.lastUsed = Time.monotonicNow()}}. We don't expose {{lastUsed}} through
constructor argument or setter, this is to avoid someone setting it to past
values. Updating of lastUsed is always done with {{Time.monotonicNow()}}.
>> Need to update the reference to BlockContainerInfo since the class is
>> removed with the patch. Otherwise, maven failed to build.
Thanks for the catch, I don't know how it got missed.
>> NIT: Line 27-33 unused imports
I don't see any unused import in ContainerMapping.java, please let me know if
I'm missing something.
>> if we always update the allocated based on the reported usage, should we
>> restrict the size to be smaller than The maximum container size (5gb)?
This is done in container report processing because in case of SCM crash the
allocated size will not be properly updated by {{ContainerStateManager}}
\[ContainerStateManager#close - will create a jira for this\], during next
start-up of SCM we don't want the allocated size to be very old (previous
successful shutdown of SCM). This update is done as a fallback mechanism in
case SCM crashes without properly updating allocated size, this is not a
correct allocated size.
In theory allocated size can exceed the container size, so I don't think we
should restrict the size here. For that matter even used size might exceed the
container size if we don't receive container reports in time (containers will
be closed while processing container reports).
>> document the new configuration key for
>> ozone.scm.container.close.threshold.percentage in ozone-default.xml?
Done.
> Ozone: SCM: Handling container report with key count and container usage.
> -
>
> Key: HDFS-12474
> URL: https://issues.apache.org/jira/browse/HDFS-12474
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Nanda kumar
> Labels: ozoneMerge
> Attachments: HDFS-12474-HDFS-7240.000.patch,
> HDFS-12474-HDFS-7240.001.patch
>
>
> Currently, the container report only contains the # of reports sent to SCM.
> We will need to provide the key count and the usage of each individual
> containers to update the SCM container state maintained by
> ContainerStateManager. This has a dependency on HDFS-12387.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12474) Ozone: SCM: Handling container report with key count and container usage.
[ https://issues.apache.org/jira/browse/HDFS-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225705#comment-16225705 ] Xiaoyu Yao commented on HDFS-12474: --- Thanks [~nandakumar131] for working on it. The patch looks good to me overall. In addition to [~linyiqun]'s comment, I just have few more comments below. ContainerMapping.java NIT: Line 27-33 unused imports Line 356-363: if we always update the allocated based on the reported usage, should we restrict the size to be smaller than The maximum container size (5gb)? SCMConfiguureKeys.java Line 226-231: document the new configuration key for ozone.scm.container.close.threshold.percentage in ozone-default.xml? Please file follow up JIRAs to persist the contain allocated size and async handling of the container report. > Ozone: SCM: Handling container report with key count and container usage. > - > > Key: HDFS-12474 > URL: https://issues.apache.org/jira/browse/HDFS-12474 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Nanda kumar > Labels: ozoneMerge > Attachments: HDFS-12474-HDFS-7240.000.patch > > > Currently, the container report only contains the # of reports sent to SCM. > We will need to provide the key count and the usage of each individual > containers to update the SCM container state maintained by > ContainerStateManager. This has a dependency on HDFS-12387. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12474) Ozone: SCM: Handling container report with key count and container usage.
[
https://issues.apache.org/jira/browse/HDFS-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224827#comment-16224827
]
Yiqun Lin commented on HDFS-12474:
--
Thanks for working on this, [~nandakumar131]. Some comments from me:
*ScmConfigKeys.java*:
line 230: Why not use {{float}} type value for the threshold-percentage
setting? And use {{Configuation#getFloat}} to get its value. The default value
here can be showed as {{0.9f}}.
*ContainerMapping.java*:
line 375: {{(containerInfo.getUsed() / containerInfo.getSize())}} should be
{{1.0 * (containerInfo.getUsed() / containerInfo.getSize())}}. The return type
of getUsed and getSize are both {{long}} type.
*ContainerStateManager.java*:
line 369: {{ContainerInfo.lastUsed}} doesn't be set here.
Would you please also add additional unit test that verify the correctness of
handling container reports?
Current change of UT is just replacing the new class.
Thanks.
> Ozone: SCM: Handling container report with key count and container usage.
> -
>
> Key: HDFS-12474
> URL: https://issues.apache.org/jira/browse/HDFS-12474
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Nanda kumar
> Labels: ozoneMerge
> Attachments: HDFS-12474-HDFS-7240.000.patch
>
>
> Currently, the container report only contains the # of reports sent to SCM.
> We will need to provide the key count and the usage of each individual
> containers to update the SCM container state maintained by
> ContainerStateManager. This has a dependency on HDFS-12387.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
