[jira] [Commented] (HDDS-128) Support for DeltaContainerReport
[
https://issues.apache.org/jira/browse/HDDS-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495544#comment-16495544
]
Nanda kumar commented on HDDS-128:
--
Thanks [~anu] for the discussion and [~xyao] for the suggestion.
I have updated the patch accordingly, please review.
> Support for DeltaContainerReport
>
>
> Key: HDDS-128
> URL: https://issues.apache.org/jira/browse/HDDS-128
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Datanode, SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-128.000.patch, HDDS-128.001.patch
>
>
> Whenever a container reaches its configured upper limit, datanode informs SCM
> about those containers through {{DeltaContainerReport}}. This jira adds
> support for {{DeltaContainerReport}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-128) Support for DeltaContainerReport
[
https://issues.apache.org/jira/browse/HDDS-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495466#comment-16495466
]
Xiaoyu Yao commented on HDDS-128:
-
Thanks [~nandakumar131] and [~anu] for the discussion. I like the 2nd proposal
better because it clearly differentiates itself from a regular container
report. Also, I propose to change the name from DeltaContainerReportsProto to
ContainerActionsProto as the former is very confusing.
{code}
message ContainerActionsProto \{
repeated ContainerAction containerActions = 1;
}
message ContainerAction { enum Action \{
CLOSE = 1;
} enum Reason \{
CONTAINER_FULL = 1;
}
required ContainerInfo container = 1;
required Action action = 2;
optional Reason reason = 3;
}
{code}
> Support for DeltaContainerReport
>
>
> Key: HDDS-128
> URL: https://issues.apache.org/jira/browse/HDDS-128
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Datanode, SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-128.000.patch
>
>
> Whenever a container reaches its configured upper limit, datanode informs SCM
> about those containers through {{DeltaContainerReport}}. This jira adds
> support for {{DeltaContainerReport}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-128) Support for DeltaContainerReport
[
https://issues.apache.org/jira/browse/HDDS-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495422#comment-16495422
]
Nanda kumar commented on HDDS-128:
--
How about this for DeltaContainerReport?
{code:java}
message DeltaContainerReportsProto {
repeated ContainerAction containerActions = 1;
}
message ContainerAction {
enum Action {
CLOSE = 1;
}
enum Reason {
CONTAINER_FULL = 1;
}
required ContainerInfo container = 1;
required Action action = 2;
optional Reason reason = 3;
}
{code}
> Support for DeltaContainerReport
>
>
> Key: HDDS-128
> URL: https://issues.apache.org/jira/browse/HDDS-128
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Datanode, SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-128.000.patch
>
>
> Whenever a container reaches its configured upper limit, datanode informs SCM
> about those containers through {{DeltaContainerReport}}. This jira adds
> support for {{DeltaContainerReport}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-128) Support for DeltaContainerReport
[
https://issues.apache.org/jira/browse/HDDS-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495408#comment-16495408
]
Anu Engineer commented on HDDS-128:
---
I am ok with Option 1: Just wondering if we should add a DatanodeRequestReason
(Maybe an Enum) to container Info itself. It will be empty most of the time,
that way containerInfo will be re-used and if this field is set, we will know
that Datanode is requesting something to SCM.
> Support for DeltaContainerReport
>
>
> Key: HDDS-128
> URL: https://issues.apache.org/jira/browse/HDDS-128
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Datanode, SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-128.000.patch
>
>
> Whenever a container reaches its configured upper limit, datanode informs SCM
> about those containers through {{DeltaContainerReport}}. This jira adds
> support for {{DeltaContainerReport}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-128) Support for DeltaContainerReport
[
https://issues.apache.org/jira/browse/HDDS-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495406#comment-16495406
]
Nanda kumar commented on HDDS-128:
--
Thanks [~anu] for the explanation, I get it. I can think of two different way
to represent the information that we want to convey to SCM through
DeltaContainerReports
1.
{code:java}
message DeltaContainerReportsProto {
repeated ContainerInfo containersToClose = 1;
}
{code}
2.
{code:java}
message DeltaContainerReportsProto {
repeated ContainerStatusProto containers = 1;
}
message ContainerStatusProto {
enum Status {
Full = 1;
}
required int64 containerID = 1;
required Status status = 2;
}
{code}
I like option 1, as it's easy to represent and understand. I'm open to other
options as well.
> Support for DeltaContainerReport
>
>
> Key: HDDS-128
> URL: https://issues.apache.org/jira/browse/HDDS-128
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Datanode, SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-128.000.patch
>
>
> Whenever a container reaches its configured upper limit, datanode informs SCM
> about those containers through {{DeltaContainerReport}}. This jira adds
> support for {{DeltaContainerReport}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-128) Support for DeltaContainerReport
[
https://issues.apache.org/jira/browse/HDDS-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495384#comment-16495384
]
Anu Engineer commented on HDDS-128:
---
[~nandakumar131] Unfortunately, I think this is one of the times that we may
need some help from Datanode. For example, if the Datanode is sending a close
container because the physical disk where this container resides is running out
of space. In other words, we might need context or some reason indicator
(something like an enum) would do.
If we don't have that then SCM's search space to puzzle out why the DN wants
this container closed is too large. It is still a request from DN – but with an
explanation for the rationale of that request. This will be useful for SCM and
for people who might be debugging the system.
> Support for DeltaContainerReport
>
>
> Key: HDDS-128
> URL: https://issues.apache.org/jira/browse/HDDS-128
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Datanode, SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-128.000.patch
>
>
> Whenever a container reaches its configured upper limit, datanode informs SCM
> about those containers through {{DeltaContainerReport}}. This jira adds
> support for {{DeltaContainerReport}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-128) Support for DeltaContainerReport
[
https://issues.apache.org/jira/browse/HDDS-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495375#comment-16495375
]
Nanda kumar commented on HDDS-128:
--
For simplicity sake, I just reused {{ContainerInfo}} structure inside
{{DeltaContainerReportsProto}}. DeltaContainerReport is sent to SCM whenever we
need to take any immediate action, based on the ContainerInfo, SCM is the one
who needs to decide what action to take. So, in this case, we (SCM) have to
check the {{ContainerInfo#used}} to make sure that the container is almost
full.
With this approach, we won't tie DeltaContainerReport to just closing the
container, in future, we can use it to take any immediate action in SCM.
DeltaContainerReport is just a subset of ContainerReport.
> Support for DeltaContainerReport
>
>
> Key: HDDS-128
> URL: https://issues.apache.org/jira/browse/HDDS-128
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Datanode, SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-128.000.patch
>
>
> Whenever a container reaches its configured upper limit, datanode informs SCM
> about those containers through {{DeltaContainerReport}}. This jira adds
> support for {{DeltaContainerReport}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-128) Support for DeltaContainerReport
[
https://issues.apache.org/jira/browse/HDDS-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495366#comment-16495366
]
Anu Engineer commented on HDDS-128:
---
[~nandakumar131] Thanks for the comment. Perhaps I am missing something, but
with this report structure how does the Datanode say "Please close container
No.5". I did not see a field that can be used to express that request.
> Support for DeltaContainerReport
>
>
> Key: HDDS-128
> URL: https://issues.apache.org/jira/browse/HDDS-128
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Datanode, SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-128.000.patch
>
>
> Whenever a container reaches its configured upper limit, datanode informs SCM
> about those containers through {{DeltaContainerReport}}. This jira adds
> support for {{DeltaContainerReport}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-128) Support for DeltaContainerReport
[
https://issues.apache.org/jira/browse/HDDS-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495362#comment-16495362
]
Nanda kumar commented on HDDS-128:
--
[~anu], yeah true. DeltaContainerReport is a high priority message than
container report but as far as proto files are concerned we don't have priority
for any messages, it's at Datanode/SCM handling of reports that makes it high
priority. This jira just introduces the message structure in proto file so that
follow up jiras can use this message.
> Support for DeltaContainerReport
>
>
> Key: HDDS-128
> URL: https://issues.apache.org/jira/browse/HDDS-128
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Datanode, SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-128.000.patch
>
>
> Whenever a container reaches its configured upper limit, datanode informs SCM
> about those containers through {{DeltaContainerReport}}. This jira adds
> support for {{DeltaContainerReport}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-128) Support for DeltaContainerReport
[
https://issues.apache.org/jira/browse/HDDS-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495354#comment-16495354
]
Anu Engineer commented on HDDS-128:
---
[~nandakumar131] Thanks for the patch. One comment.
we have to keep in mind is that Delta Container Reports is the “Mechanism” for
the Datanode to tell SCM to close a “Container”. It can happen due to many
conditions
The Container is getting full.
The Disk is running out of space.
In this patcch, Delta Container report is a report on the mini-status on a set
of containers , but I tend to think of this as a high priority message from
Datanode.
> Support for DeltaContainerReport
>
>
> Key: HDDS-128
> URL: https://issues.apache.org/jira/browse/HDDS-128
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Datanode, SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-128.000.patch
>
>
> Whenever a container reaches its configured upper limit, datanode informs SCM
> about those containers through {{DeltaContainerReport}}. This jira adds
> support for {{DeltaContainerReport}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-128) Support for DeltaContainerReport
[
https://issues.apache.org/jira/browse/HDDS-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494306#comment-16494306
]
genericqa commented on HDDS-128:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 32m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
44m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 1s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
28s{color} | {color:green} container-service in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
27s{color} | {color:red} The patch generated 10 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDDS-128 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12925464/HDDS-128.000.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 7d0b78e79341 3.13.0-141-generic #190-Ubuntu SMP Fri Jan 19
12:52:38 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 201440b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/206/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HDDS-Build/206/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 325 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/container-service U: hadoop-hdds/container-service |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/206/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Support for DeltaContainerReport
>
>
> Key: HDDS-128
> URL: https://issues.apache.org/jira/browse/HDDS-128
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Datanode, SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-128.000.patch
>
>
> Whenever a container reaches its configured upper limit, datanode informs SCM
> about those containers through {{DeltaContainerReport}}. This jira adds
> support for {{DeltaContainerReport}}.
--
This message was sent by
