[jira] [Commented] (HDDS-1511) Space tracking for Open Containers in HDDS Volumes
[ https://issues.apache.org/jira/browse/HDDS-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840604#comment-16840604 ] Hudson commented on HDDS-1511: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16553 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16553/]) HDDS-1511. Space tracking for Open Containers in HDDS Volumes. (arp7: rev 9569015802e695f1c242c74d5ac9df27e180374c) * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerPersistence.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerData.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerSet.java > Space tracking for Open Containers in HDDS Volumes > -- > > Key: HDDS-1511 > URL: https://issues.apache.org/jira/browse/HDDS-1511 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: HDDS-1511.000.patch, HDDS-1511.001.patch > > Time Spent: 2h 50m > Remaining Estimate: 0h > > For every HDDS Volume, track the space usage in open containers. Introduce a > counter committedBytes in HddsVolume - this counts the remaining space in > Open containers until they reach max capacity. The counter is incremented (by > container max capacity) for every container create. And decremented (by chunk > size) for every chunk write. > Space tracking for open containers will enable adding a safety check during > container create. > If there is not sufficient free space in the volume, the container create > operation can be failed. > The scope of this jira is to just add the space tracking for Open Containers. > Checking for space and failing container create will be introduced in a > subsequent jira. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1511) Space tracking for Open Containers in HDDS Volumes
[
https://issues.apache.org/jira/browse/HDDS-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836962#comment-16836962
]
Hadoop QA commented on HDDS-1511:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 4m
12s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
27s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 26s{color} | {color:orange} hadoop-hdds: The patch generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 12s{color}
| {color:red} hadoop-hdds in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 39s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdds.scm.pipeline.TestRatisPipelineProvider |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/PreCommit-HDDS-Build/2683/artifact/out/Dockerfile
|
| JIRA Issue | HDDS-1511 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12968355/HDDS-1511.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
[jira] [Commented] (HDDS-1511) Space tracking for Open Containers in HDDS Volumes
[ https://issues.apache.org/jira/browse/HDDS-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836892#comment-16836892 ] Supratim Deka commented on HDDS-1511: - addressed comment from [~arpitagarwal] in patch 001. will add a pull request as well. > Space tracking for Open Containers in HDDS Volumes > -- > > Key: HDDS-1511 > URL: https://issues.apache.org/jira/browse/HDDS-1511 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > Labels: pull-request-available > Attachments: HDDS-1511.000.patch, HDDS-1511.001.patch > > Time Spent: 10m > Remaining Estimate: 0h > > For every HDDS Volume, track the space usage in open containers. Introduce a > counter committedBytes in HddsVolume - this counts the remaining space in > Open containers until they reach max capacity. The counter is incremented (by > container max capacity) for every container create. And decremented (by chunk > size) for every chunk write. > Space tracking for open containers will enable adding a safety check during > container create. > If there is not sufficient free space in the volume, the container create > operation can be failed. > The scope of this jira is to just add the space tracking for Open Containers. > Checking for space and failing container create will be introduced in a > subsequent jira. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1511) Space tracking for Open Containers in HDDS Volumes
[
https://issues.apache.org/jira/browse/HDDS-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836829#comment-16836829
]
Arpit Agarwal commented on HDDS-1511:
-
Thanks for the patch [~sdeka].
{code}
// reduce space committed for this container equal to bytes written
if (committedSpace) {
this.getVolume().incCommittedBytes(0 - bytes);
}
{code}
If the container grows beyond its max size which can happen, then we can
potentially subtract more than was originally committed. So we should probably
check for this case. Perhaps with something like:
{code}
if (committedSpace && getWriteBytes() < getMaxSize()) {
this.getVolume().incCommittedBytes(0 - bytes);
{code}
We may have to think of the synchronization. We also need the same fix in
{{releaseCommitSpace}}.
Also committedBytes is not currently subtracted from the available space. I
assume you will do that in a separate patch.
> Space tracking for Open Containers in HDDS Volumes
> --
>
> Key: HDDS-1511
> URL: https://issues.apache.org/jira/browse/HDDS-1511
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode
>Reporter: Supratim Deka
>Assignee: Supratim Deka
>Priority: Major
> Attachments: HDDS-1511.000.patch
>
>
> For every HDDS Volume, track the space usage in open containers. Introduce a
> counter committedBytes in HddsVolume - this counts the remaining space in
> Open containers until they reach max capacity. The counter is incremented (by
> container max capacity) for every container create. And decremented (by chunk
> size) for every chunk write.
> Space tracking for open containers will enable adding a safety check during
> container create.
> If there is not sufficient free space in the volume, the container create
> operation can be failed.
> The scope of this jira is to just add the space tracking for Open Containers.
> Checking for space and failing container create will be introduced in a
> subsequent jira.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1511) Space tracking for Open Containers in HDDS Volumes
[
https://issues.apache.org/jira/browse/HDDS-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836776#comment-16836776
]
Hadoop QA commented on HDDS-1511:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 18m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
32m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
28s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
27s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 37m
36s{color} | {color:red} root in trunk has 6 extant findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} branch/hadoop-ozone/integration-test no findbugs
output file (findbugsXml.xml) {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 38s{color} | {color:orange} root: The patch generated 3 new + 0 unchanged -
0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 17m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
20s{color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} hadoop-ozone/integration-test has no data from
findbugs {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}149m 25s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 1m
5s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}406m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.container.common.impl.TestContainerSet |
| | hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce
[jira] [Commented] (HDDS-1511) Space tracking for Open Containers in HDDS Volumes
[ https://issues.apache.org/jira/browse/HDDS-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836659#comment-16836659 ] Arpit Agarwal commented on HDDS-1511: - bq. Perhaps a better investment is to write code that handles the container creation or chunk write failure, or come up with an algorithm to look at what HDFS is doing too. [~anu], you are absolutely right. We should and we will make sure we robustly handle IO failures. This particular Jira fixes a very limited scenario of putting new containers on obviously full volumes. It is not a substitute for robust failure handling. > Space tracking for Open Containers in HDDS Volumes > -- > > Key: HDDS-1511 > URL: https://issues.apache.org/jira/browse/HDDS-1511 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > Attachments: HDDS-1511.000.patch > > > For every HDDS Volume, track the space usage in open containers. Introduce a > counter committedBytes in HddsVolume - this counts the remaining space in > Open containers until they reach max capacity. The counter is incremented (by > container max capacity) for every container create. And decremented (by chunk > size) for every chunk write. > Space tracking for open containers will enable adding a safety check during > container create. > If there is not sufficient free space in the volume, the container create > operation can be failed. > The scope of this jira is to just add the space tracking for Open Containers. > Checking for space and failing container create will be introduced in a > subsequent jira. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1511) Space tracking for Open Containers in HDDS Volumes
[ https://issues.apache.org/jira/browse/HDDS-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836546#comment-16836546 ] Anu Engineer commented on HDDS-1511: Is this part of the I/O handling work? [~arpitagarwal] > Space tracking for Open Containers in HDDS Volumes > -- > > Key: HDDS-1511 > URL: https://issues.apache.org/jira/browse/HDDS-1511 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > Attachments: HDDS-1511.000.patch > > > For every HDDS Volume, track the space usage in open containers. Introduce a > counter committedBytes in HddsVolume - this counts the remaining space in > Open containers until they reach max capacity. The counter is incremented (by > container max capacity) for every container create. And decremented (by chunk > size) for every chunk write. > Space tracking for open containers will enable adding a safety check during > container create. > If there is not sufficient free space in the volume, the container create > operation can be failed. > The scope of this jira is to just add the space tracking for Open Containers. > Checking for space and failing container create will be introduced in a > subsequent jira. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1511) Space tracking for Open Containers in HDDS Volumes
[ https://issues.apache.org/jira/browse/HDDS-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836547#comment-16836547 ] Anu Engineer commented on HDDS-1511: [~sdeka] Not sure how to handle this issue, Thought I will point it out. HDFS will be sharing these disks. So tracking free space based on Open containers will not address the use case you are mentioning. bq. Space tracking for open containers will enable adding a safety check during container create. If there is not sufficient free space in the volume, the container create operation can be failed. Perhaps a better investment is to write code that handles the container creation or chunk write failure, or come up with an algorithm to look at what HDFS is doing too. > Space tracking for Open Containers in HDDS Volumes > -- > > Key: HDDS-1511 > URL: https://issues.apache.org/jira/browse/HDDS-1511 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > Attachments: HDDS-1511.000.patch > > > For every HDDS Volume, track the space usage in open containers. Introduce a > counter committedBytes in HddsVolume - this counts the remaining space in > Open containers until they reach max capacity. The counter is incremented (by > container max capacity) for every container create. And decremented (by chunk > size) for every chunk write. > Space tracking for open containers will enable adding a safety check during > container create. > If there is not sufficient free space in the volume, the container create > operation can be failed. > The scope of this jira is to just add the space tracking for Open Containers. > Checking for space and failing container create will be introduced in a > subsequent jira. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
