[jira] [Commented] (YARN-10493) RunC container repository v2
[
https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397576#comment-17397576
]
Hadoop QA commented on YARN-10493:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
46s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue} 0m
1s{color} | | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 3 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 12m
29s{color} | | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
2s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
29s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
16s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
42s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
59s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
42s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
33s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 5m
29s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 7s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
53s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
53s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
12s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
12s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
37s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
42s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
27s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
20s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 5m
46s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} |
[jira] [Commented] (YARN-10493) RunC container repository v2
[
https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17342058#comment-17342058
]
Hadoop QA commented on YARN-10493:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue} 0m
1s{color} | | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 3 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 16m
10s{color} | | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
18s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
19s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
22s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
51s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
42s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
15s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
26s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 5m
35s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 21s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
21s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
20s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
20s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
42s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
42s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
57s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
47s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
22s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
27s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 7m
1s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} |
[jira] [Commented] (YARN-10493) RunC container repository v2
[
https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17333560#comment-17333560
]
Eric Badger commented on YARN-10493:
bq. In theory we could change that if there is a benefit in your opinion, but
my initial reaction is that adding sub-directories to that namespace may make
it harder to track images (cleanup, governance, perhaps even quotas, etc.).
I don't think it's a huge deal. A nice to have feature, but if it requires a
major rework then I don't think it's necessary. The reason I think it would be
nice is so that we can more cleanly segment our images. E.g. you could have
{{hadoop/small-image/rhel7:7.9}} or something like that. But again, it's not a
huge deal if it's difficult
> RunC container repository v2
>
>
> Key: YARN-10493
> URL: https://issues.apache.org/jira/browse/YARN-10493
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, yarn
>Affects Versions: 3.3.0
>Reporter: Craig Condit
>Assignee: Matthew Sharp
>Priority: Major
> Labels: pull-request-available
> Attachments: runc-container-repository-v2-design.pdf,
> runc-container-repository-v2-design_updated.pdf
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The current runc container repository design has scalability and usability
> issues which will likely limit widespread adoption. We should address this
> with a new, V2 layout.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10493) RunC container repository v2
[ https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17333457#comment-17333457 ] Matthew Sharp commented on YARN-10493: -- An exception is thrown for that to ensure it matches our storage format for the HDFS properties file. This is the same logic as the Java CLI import tool from YARN-10494. Example HDFS Path: /runc-root/meta/[namespace]/@.properties In theory we could change that if there is a benefit in your opinion, but my initial reaction is that adding sub-directories to that namespace may make it harder to track images (cleanup, governance, perhaps even quotas, etc.). > RunC container repository v2 > > > Key: YARN-10493 > URL: https://issues.apache.org/jira/browse/YARN-10493 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, yarn >Affects Versions: 3.3.0 >Reporter: Craig Condit >Assignee: Matthew Sharp >Priority: Major > Labels: pull-request-available > Attachments: runc-container-repository-v2-design.pdf, > runc-container-repository-v2-design_updated.pdf > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The current runc container repository design has scalability and usability > issues which will likely limit widespread adoption. We should address this > with a new, V2 layout. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10493) RunC container repository v2
[
https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1773#comment-1773
]
Eric Badger commented on YARN-10493:
What I'm saying on the split thing is that in the current state
{{hadoop/rhel7/myimage:current}} would throw an exception. But I don't see why
that is necessary. In the above case, why not have {{hadoop}} as the namespace
and {{rhel7/myimage:current}} as the image name?
> RunC container repository v2
>
>
> Key: YARN-10493
> URL: https://issues.apache.org/jira/browse/YARN-10493
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, yarn
>Affects Versions: 3.3.0
>Reporter: Craig Condit
>Assignee: Matthew Sharp
>Priority: Major
> Labels: pull-request-available
> Attachments: runc-container-repository-v2-design.pdf,
> runc-container-repository-v2-design_updated.pdf
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The current runc container repository design has scalability and usability
> issues which will likely limit widespread adoption. We should address this
> with a new, V2 layout.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10493) RunC container repository v2
[ https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17333276#comment-17333276 ] Matthew Sharp commented on YARN-10493: -- Thanks for the review. I can submit an update for the feedback. The string split portion with a negative limit will actually split at the delimiter as many times as possible. I do this to help account for invalid input that otherwise would crash the NodeManager if it isn't caught. A lot of the new test cases added try to account for various examples of accidental typos that could cause issues. For the dependency on YARN-10494 I can start to look at that next. > RunC container repository v2 > > > Key: YARN-10493 > URL: https://issues.apache.org/jira/browse/YARN-10493 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, yarn >Affects Versions: 3.3.0 >Reporter: Craig Condit >Assignee: Matthew Sharp >Priority: Major > Labels: pull-request-available > Attachments: runc-container-repository-v2-design.pdf, > runc-container-repository-v2-design_updated.pdf > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The current runc container repository design has scalability and usability > issues which will likely limit widespread adoption. We should address this > with a new, V2 layout. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10493) RunC container repository v2
[
https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17326150#comment-17326150
]
Eric Badger commented on YARN-10493:
Thanks for the latest patch. I tested out the patch along with the CLI tool
from YARN-10494 and everything seems to be working well. The addition of
namespaces has fixed my issue from last time with the {{hadoop/rhel7:current}}
image. But I do have a few comments. In addition to these comments, I would
like to commit both this JIRA and YARN-10494 at the same time, because I don't
particularly think either makes sense without the other. And YARN-10494 has a
blocker on it because of the docker->overlayfs incompatibilities with
whiteout/opaque files. Anyway, here are my comments on this JIRA
{noformat}
if (!tag.equals("")) {
{noformat}
nit: There are a few times where the patch uses this, but I think {{isEmpty()}}
is more appropriate than {{equals("")}}.
{noformat}
String[] nameParts = imageCoordinates.split("/", -1);
String imageTag;
if (nameParts.length == 2) {
metaNamespace = nameParts[0];
imageTag = nameParts[1];
} else if (nameParts.length == 1) {
imageTag = nameParts[0];
} else {
throw new IllegalArgumentException("Invalid image coordinates: "
+ imageCoordinates);
}
{noformat}
According to the
[documentation|https://docs.oracle.com/javase/8/docs/api/java/lang/String.html#split-java.lang.String-int-]
for {{split}}, this code will create a String array with the number of
elements equal to the number of {{/}} + 1. But then we only look at the first 2
parts of the array. Is there a reason not to take only the part before the
first slash as the namespace and then take the rest as the imageTag?
> RunC container repository v2
>
>
> Key: YARN-10493
> URL: https://issues.apache.org/jira/browse/YARN-10493
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, yarn
>Affects Versions: 3.3.0
>Reporter: Craig Condit
>Assignee: Matthew Sharp
>Priority: Major
> Labels: pull-request-available
> Attachments: runc-container-repository-v2-design.pdf,
> runc-container-repository-v2-design_updated.pdf
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The current runc container repository design has scalability and usability
> issues which will likely limit widespread adoption. We should address this
> with a new, V2 layout.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10493) RunC container repository v2
[ https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313424#comment-17313424 ] Matthew Sharp commented on YARN-10493: -- [~ebadger] The latest PR contains the namespace support that we had discussed. I also updated the design doc to outline that a bit more. > RunC container repository v2 > > > Key: YARN-10493 > URL: https://issues.apache.org/jira/browse/YARN-10493 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, yarn >Affects Versions: 3.3.0 >Reporter: Craig Condit >Assignee: Matthew Sharp >Priority: Major > Labels: pull-request-available > Attachments: runc-container-repository-v2-design.pdf, > runc-container-repository-v2-design_updated.pdf > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The current runc container repository design has scalability and usability > issues which will likely limit widespread adoption. We should address this > with a new, V2 layout. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10493) RunC container repository v2
[
https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313418#comment-17313418
]
Hadoop QA commented on YARN-10493:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
40s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue} 0m
1s{color} | | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 3 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 14m
53s{color} | | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
9s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
5s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
54s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
47s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
51s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
28s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
39s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 5m
32s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 44s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
29s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
29s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
46s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
46s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
1s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
35s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
12s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
25s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 5m
52s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} |
[jira] [Commented] (YARN-10493) RunC container repository v2
[
https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313322#comment-17313322
]
Hadoop QA commented on YARN-10493:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue} 0m
1s{color} | | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 3 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 14m
39s{color} | | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
10s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
7s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
53s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
43s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
53s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
29s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
38s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 5m
36s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 1s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
33s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
33s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
48s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
48s{color} | | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} blanks {color} | {color:red} 0m
0s{color} |
[/blanks-eol.txt|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2848/1/artifact/out/blanks-eol.txt]
| {color:red} The patch has 1 line(s) that end in blanks. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
36s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
16s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
24s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
|
[jira] [Commented] (YARN-10493) RunC container repository v2
[ https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308201#comment-17308201 ] Matthew Sharp commented on YARN-10493: -- Good news for the test. I can add that to the improvement list for the YARN-10494 tool, since that will break the localizer. I have a working patch the namespaces, going to test a bit more on our cluster tomorrow and then I will update the PR and update the pdf. > RunC container repository v2 > > > Key: YARN-10493 > URL: https://issues.apache.org/jira/browse/YARN-10493 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, yarn >Affects Versions: 3.3.0 >Reporter: Craig Condit >Assignee: Matthew Sharp >Priority: Major > Labels: pull-request-available > Attachments: runc-container-repository-v2-design.pdf > > Time Spent: 0.5h > Remaining Estimate: 0h > > The current runc container repository design has scalability and usability > issues which will likely limit widespread adoption. We should address this > with a new, V2 layout. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10493) RunC container repository v2
[ https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308180#comment-17308180 ] Eric Badger commented on YARN-10493: I did have a weird umask set. Reverting back to the default umask fixed the localizer errors that I posted above. However, the tool should probably explicitly specify 755 and 644 perms for the directories and files respectively > RunC container repository v2 > > > Key: YARN-10493 > URL: https://issues.apache.org/jira/browse/YARN-10493 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, yarn >Affects Versions: 3.3.0 >Reporter: Craig Condit >Assignee: Matthew Sharp >Priority: Major > Labels: pull-request-available > Attachments: runc-container-repository-v2-design.pdf > > Time Spent: 0.5h > Remaining Estimate: 0h > > The current runc container repository design has scalability and usability > issues which will likely limit widespread adoption. We should address this > with a new, V2 layout. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10493) RunC container repository v2
[ https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308173#comment-17308173 ] Eric Badger commented on YARN-10493: Hmm, must be a default umask issue or something on my testing environment. After fixing the perms and a small container-executor.cfg issue, I've been able to successfully run a sleep job using the V2 plugins! > RunC container repository v2 > > > Key: YARN-10493 > URL: https://issues.apache.org/jira/browse/YARN-10493 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, yarn >Affects Versions: 3.3.0 >Reporter: Craig Condit >Assignee: Matthew Sharp >Priority: Major > Labels: pull-request-available > Attachments: runc-container-repository-v2-design.pdf > > Time Spent: 0.5h > Remaining Estimate: 0h > > The current runc container repository design has scalability and usability > issues which will likely limit widespread adoption. We should address this > with a new, V2 layout. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10493) RunC container repository v2
[ https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308172#comment-17308172 ] Matthew Sharp commented on YARN-10493: -- No I don't see that in our cluster. We have 755 perms on the directories and 644 on the files it looks like. I actually just recently deleted the old /runc-root as well with the v2 change for testing and it recreated with default umask I am assuming. For the CLI tool imports that is running as our hdfs admin user still (not ideal for now), but for testing seems fine. > RunC container repository v2 > > > Key: YARN-10493 > URL: https://issues.apache.org/jira/browse/YARN-10493 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, yarn >Affects Versions: 3.3.0 >Reporter: Craig Condit >Assignee: Matthew Sharp >Priority: Major > Labels: pull-request-available > Attachments: runc-container-repository-v2-design.pdf > > Time Spent: 0.5h > Remaining Estimate: 0h > > The current runc container repository design has scalability and usability > issues which will likely limit widespread adoption. We should address this > with a new, V2 layout. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10493) RunC container repository v2
[
https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308169#comment-17308169
]
Eric Badger commented on YARN-10493:
{noformat}
[ebadger@foo hadoop]$ hadoop fs -ls /runc-root/*/*/*
WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of
HADOOP_PREFIX.
-rw--- 10 ebadger supergroup 11166 2021-03-24 00:06
/runc-root/config/a9/a9a241e617577cf0da93c89010d0026de8327c8220c732f2ede29d2ce15588cf
-rw--- 10 ebadger supergroup 4096 2021-03-24 00:06
/runc-root/layer/30/30e0953848a62b1477a41e6c09d2f781ace3e6cfb215f01be9dacabdd9480d90.sqsh
-rw--- 10 ebadger supergroup156 2021-03-24 00:06
/runc-root/layer/30/30e0953848a62b1477a41e6c09d2f781ace3e6cfb215f01be9dacabdd9480d90.tar.gz
-rw--- 10 ebadger supergroup 4096 2021-03-24 00:06
/runc-root/layer/71/71a0a977369767160e5ed5f8ab279558240eb19efa880725049c4651411c2225.sqsh
-rw--- 10 ebadger supergroup185 2021-03-24 00:06
/runc-root/layer/71/71a0a977369767160e5ed5f8ab279558240eb19efa880725049c4651411c2225.tar.gz
-rw--- 10 ebadger supergroup 26095616 2021-03-24 00:06
/runc-root/layer/72/726141ff510fe8ee7d540faa490649332a561f79ce9b5d02045f7e0db5e4cfbc.sqsh
-rw--- 10 ebadger supergroup 26687036 2021-03-24 00:06
/runc-root/layer/72/726141ff510fe8ee7d540faa490649332a561f79ce9b5d02045f7e0db5e4cfbc.tar.gz
-rw--- 10 ebadger supergroup 12288 2021-03-24 00:06
/runc-root/layer/8c/8c4f37442a65ac28bb23c8a0c408f1f2c061b8928abfaf8a40050ebae6130974.sqsh
-rw--- 10 ebadger supergroup 12057 2021-03-24 00:06
/runc-root/layer/8c/8c4f37442a65ac28bb23c8a0c408f1f2c061b8928abfaf8a40050ebae6130974.tar.gz
-rw--- 10 ebadger supergroup 98697216 2021-03-24 00:06
/runc-root/layer/98/98dc2361422a32ef978770b879dd1d0079242cc55980cfd2205939a6796d309f.sqsh
-rw--- 10 ebadger supergroup 99451719 2021-03-24 00:06
/runc-root/layer/98/98dc2361422a32ef978770b879dd1d0079242cc55980cfd2205939a6796d309f.tar.gz
-rw--- 10 ebadger supergroup 121638912 2021-03-24 00:06
/runc-root/layer/d5/d5eed6e7343938620cc22b9b2db0d6a77e8de76ed1387e7b7288d034fd86a92d.sqsh
-rw--- 10 ebadger supergroup 123724262 2021-03-24 00:06
/runc-root/layer/d5/d5eed6e7343938620cc22b9b2db0d6a77e8de76ed1387e7b7288d034fd86a92d.tar.gz
-rw--- 10 ebadger supergroup 205000704 2021-03-24 00:06
/runc-root/layer/f6/f64d0ed7e0de1a2c56f69eaae5cfe8351b23cc2d600cc226557881862a5bdf5f.sqsh
-rw--- 10 ebadger supergroup 205322058 2021-03-24 00:06
/runc-root/layer/f6/f64d0ed7e0de1a2c56f69eaae5cfe8351b23cc2d600cc226557881862a5bdf5f.tar.gz
-rw--- 10 ebadger supergroup 1791 2021-03-24 00:06
/runc-root/manifest/f8/f849453b22d5e6a2e2f1390dc021cd2a786bcd923fffa9e778f3be6c87a0d3fe
-rw--- 10 ebadger supergroup236 2021-03-24 00:06
/runc-root/meta/hadoop/rhel7@current.properties
-rw--- 10 ebadger supergroup236 2021-03-24 00:15
/runc-root/meta/hadoop/rhel7@latest.properties
-rw--- 10 ebadger supergroup236 2021-03-24 20:21
/runc-root/meta/library/rhel7@current.properties
{noformat}
For reference, the perms on all of the files are 600.
> RunC container repository v2
>
>
> Key: YARN-10493
> URL: https://issues.apache.org/jira/browse/YARN-10493
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, yarn
>Affects Versions: 3.3.0
>Reporter: Craig Condit
>Assignee: Matthew Sharp
>Priority: Major
> Labels: pull-request-available
> Attachments: runc-container-repository-v2-design.pdf
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The current runc container repository design has scalability and usability
> issues which will likely limit widespread adoption. We should address this
> with a new, V2 layout.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10493) RunC container repository v2
[
https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308167#comment-17308167
]
Eric Badger commented on YARN-10493:
{noformat}
2021-03-24 20:21:56,225 WARN [Public Localizer]
localizer.LocalResourcesTrackerImpl
(LocalResourcesTrackerImpl.java:handle(184)) - Received LOCALIZATION_FAILED
event for request {
/runc-root/layer/f6/f64d0ed7e0de1a2c56f69eaae5cfe8351b23cc2d600cc226557881862a5bdf5f.sqsh,
1616544364679, FILE, null } but localized resource is missing
2021-03-24 20:21:56,226 ERROR [Public Localizer]
localizer.ResourceLocalizationService
(ResourceLocalizationService.java:run(1004)) - Failed to download resource { {
/runc-root/layer/f6/f64d0ed7e0de1a2c56f69eaae5cfe8351b23cc2d600cc226557881862a5bdf5f.sqsh,
1616544364679, FILE, null },pending,[],50806868248172187,DOWNLOADING}
java.io.IOException: Resource
/runc-root/layer/f6/f64d0ed7e0de1a2c56f69eaae5cfe8351b23cc2d600cc226557881862a5bdf5f.sqsh
is not publicly accessible and as such cannot be part of the public cache.
2021-03-24 20:21:56,227 WARN [Public Localizer]
localizer.LocalResourcesTrackerImpl
(LocalResourcesTrackerImpl.java:handle(184)) - Received LOCALIZATION_FAILED
event for request {
/runc-root/layer/d5/d5eed6e7343938620cc22b9b2db0d6a77e8de76ed1387e7b7288d034fd86a92d.sqsh,
1616544363068, FILE, null } but localized resource is missing
2021-03-24 20:21:56,227 ERROR [Public Localizer]
localizer.ResourceLocalizationService
(ResourceLocalizationService.java:run(1004)) - Failed to download resource { {
/runc-root/layer/d5/d5eed6e7343938620cc22b9b2db0d6a77e8de76ed1387e7b7288d034fd86a92d.sqsh,
1616544363068, FILE, null },pending,[],50806868248209381,DOWNLOADING}
java.io.IOException: Resource
/runc-root/layer/d5/d5eed6e7343938620cc22b9b2db0d6a77e8de76ed1387e7b7288d034fd86a92d.sqsh
is not publicly accessible and as such cannot be part of the public cache.
2021-03-24 20:21:56,227 WARN [Public Localizer]
localizer.LocalResourcesTrackerImpl
(LocalResourcesTrackerImpl.java:handle(184)) - Received LOCALIZATION_FAILED
event for request {
/runc-root/layer/30/30e0953848a62b1477a41e6c09d2f781ace3e6cfb215f01be9dacabdd9480d90.sqsh,
1616544367971, FILE, null } but localized resource is missing
2021-03-24 20:21:56,227 ERROR [Public Localizer]
localizer.ResourceLocalizationService
(ResourceLocalizationService.java:run(1004)) - Failed to download resource { {
/runc-root/layer/30/30e0953848a62b1477a41e6c09d2f781ace3e6cfb215f01be9dacabdd9480d90.sqsh,
1616544367971, FILE, null },pending,[],50806868248366695,DOWNLOADING}
java.io.IOException: Resource
/runc-root/layer/30/30e0953848a62b1477a41e6c09d2f781ace3e6cfb215f01be9dacabdd9480d90.sqsh
is not publicly accessible and as such cannot be part of the public cache.
2021-03-24 20:21:56,228 WARN [Public Localizer]
localizer.LocalResourcesTrackerImpl
(LocalResourcesTrackerImpl.java:handle(184)) - Received LOCALIZATION_FAILED
event for request {
/runc-root/layer/71/71a0a977369767160e5ed5f8ab279558240eb19efa880725049c4651411c2225.sqsh,
1616544367125, FILE, null } but localized resource is missing
2021-03-24 20:21:56,228 ERROR [Public Localizer]
localizer.ResourceLocalizationService
(ResourceLocalizationService.java:run(1004)) - Failed to download resource { {
/runc-root/layer/71/71a0a977369767160e5ed5f8ab279558240eb19efa880725049c4651411c2225.sqsh,
1616544367125, FILE, null },pending,[],50806868248376369,DOWNLOADING}
java.io.IOException: Resource
/runc-root/layer/71/71a0a977369767160e5ed5f8ab279558240eb19efa880725049c4651411c2225.sqsh
is not publicly accessible and as such cannot be part of the public cache.
2021-03-24 20:21:56,228 WARN [Public Localizer]
localizer.LocalResourcesTrackerImpl
(LocalResourcesTrackerImpl.java:handle(184)) - Received LOCALIZATION_FAILED
event for request {
/runc-root/config/a9/a9a241e617577cf0da93c89010d0026de8327c8220c732f2ede29d2ce15588cf,
1616544368404, FILE, null } but localized resource is missing
{noformat}
Building and running with an image under the default "library" namespace, I run
into these permission errors. The errors are pretty clear, but is this
something that you run into in your production environments? Is there a setup
step that is necessary before or after running the CLI tool to fix the perms on
the files?
> RunC container repository v2
>
>
> Key: YARN-10493
> URL: https://issues.apache.org/jira/browse/YARN-10493
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, yarn
>Affects Versions: 3.3.0
>Reporter: Craig Condit
>Assignee: Matthew Sharp
>Priority: Major
> Labels: pull-request-available
> Attachments: runc-container-repository-v2-design.pdf
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The current runc container
[jira] [Commented] (YARN-10493) RunC container repository v2
[ https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308165#comment-17308165 ] Eric Badger commented on YARN-10493: Yea, I think that would be a good improvement to the plugin implementation. > RunC container repository v2 > > > Key: YARN-10493 > URL: https://issues.apache.org/jira/browse/YARN-10493 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, yarn >Affects Versions: 3.3.0 >Reporter: Craig Condit >Assignee: Matthew Sharp >Priority: Major > Labels: pull-request-available > Attachments: runc-container-repository-v2-design.pdf > > Time Spent: 0.5h > Remaining Estimate: 0h > > The current runc container repository design has scalability and usability > issues which will likely limit widespread adoption. We should address this > with a new, V2 layout. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10493) RunC container repository v2
[
https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307897#comment-17307897
]
Matthew Sharp commented on YARN-10493:
--
I looked at YARN-10494 (CLI Tool) design and that was the missing piece. I
originally included a static directory you can set for a single namespace
(defaults to library if you don't). But the CLI tool design pdf and your usage
example makes more sense, to make that user configurable and only default to a
namespace if that is not provided. I can work on an update to support that and
also add to the pdf design to make that more clear.
YARN_CONTAINER_RUNTIME_RUNC_IMAGE=[namespace]/[:tag]
Examples:
{code:java}
YARN_CONTAINER_RUNTIME_RUNC_IMAGE=busybox
/runc-root/meta/library/busybox@latest.properties{code}
{code:java}
YARN_CONTAINER_RUNTIME_RUNC_IMAGE=hadoop/busybox:latest
/runc-root/meta/hadoop/busybox@latest.properties
{code}
{code:java}
YARN_CONTAINER_RUNTIME_RUNC_IMAGE=shared/busybox:1.0.0
/runc-root/meta/shared/busybox@1.0.0.properties
{code}
> RunC container repository v2
>
>
> Key: YARN-10493
> URL: https://issues.apache.org/jira/browse/YARN-10493
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, yarn
>Affects Versions: 3.3.0
>Reporter: Craig Condit
>Assignee: Matthew Sharp
>Priority: Major
> Labels: pull-request-available
> Attachments: runc-container-repository-v2-design.pdf
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The current runc container repository design has scalability and usability
> issues which will likely limit widespread adoption. We should address this
> with a new, V2 layout.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10493) RunC container repository v2
[
https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307481#comment-17307481
]
Eric Badger commented on YARN-10493:
Additionally, I've run into some issues while testing.
{noformat:title=CLI Invocation}
hadoop jar ./hadoop-tools/hadoop-runc/target/hadoop-runc-3.4.0-SNAPSHOT.jar
org.apache.hadoop.runc.tools.ImportDockerImage -r docker.foobar.com:
hadoop-images/hadoop/rhel7 hadoop/rhel7
{noformat}
{noformat}
[ebadger@foo hadoop]$ hadoop fs -ls
/runc-root/meta/hadoop/rhel7@latest.properties
-rw--- 10 ebadger supergroup236 2021-03-24 00:15
/runc-root/meta/hadoop/rhel7@latest.properties
{noformat}
Here's the properties file after the CLI tool completes.
{noformat}
yarn.nodemanager.runtime.linux.runc.image-tag-to-manifest-plugin
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.runc.ImageTagToManifestV2Plugin
yarn.nodemanager.runtime.linux.runc.manifest-to-resources-plugin
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.runc.HdfsManifestToResourcesV2Plugin
{noformat}
Then I set these properties as well as adding {{runc}} to the allowed-runtimes
config.
{noformat}
export
vars="YARN_CONTAINER_RUNTIME_TYPE=runc,YARN_CONTAINER_RUNTIME_RUNC_IMAGE=hadoop/rhel7";
$HADOOP_HOME/bin/hadoop jar
$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.*-tests.jar
sleep -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME=$HADOOP_HOME"
-Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME=$HADOOP_HOME"
-Dyarn.app.mapreduce.am.env=$vars -Dmapreduce.map.env=$vars
-Dmapreduce.reduce.env=$vars -mt 1 -rt 1 -m 1 -r 1
{noformat}
I ran a sleep job using this command.
{noformat}
2021-03-24 00:26:07,823 DEBUG [NM ContainerManager dispatcher]
runc.ImageTagToManifestV2Plugin
(ImageTagToManifestV2Plugin.java:getHdfsImageToHashReader(144)) - Checking HDFS
for image file: /runc-root/meta/library/hadoop/rhel7@latest.properties
2021-03-24 00:26:07,825 WARN [NM ContainerManager dispatcher]
runc.ImageTagToManifestV2Plugin
(ImageTagToManifestV2Plugin.java:getHdfsImageToHashReader(148)) - Did not load
the hdfs image to hash properties file, file doesn't exist
2021-03-24 00:26:07,828 WARN [NM ContainerManager dispatcher]
container.ContainerImpl (ContainerImpl.java:transition(1261)) - Failed to parse
resource-request
java.io.FileNotFoundException: File does not exist:
/runc-root/manifest/ha/hadoop/rhel7
{noformat}
Then I got this error in the NM when it was trying to resolve the tag. It added
the default {{metaNamespaceDir}} (which is library) into the path when looking
for the properties file. But when the CLI tool ran, it didn't add the
{{metaNamespaceDir}}. I didn't have the config set in my configs at all, so the
NM was using the conf default.
I'm not sure if I did anything wrong here or not, but it seems inconsistent to
me. Let me know what you think
> RunC container repository v2
>
>
> Key: YARN-10493
> URL: https://issues.apache.org/jira/browse/YARN-10493
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, yarn
>Affects Versions: 3.3.0
>Reporter: Craig Condit
>Assignee: Matthew Sharp
>Priority: Major
> Labels: pull-request-available
> Attachments: runc-container-repository-v2-design.pdf
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The current runc container repository design has scalability and usability
> issues which will likely limit widespread adoption. We should address this
> with a new, V2 layout.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10493) RunC container repository v2
[ https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307467#comment-17307467 ] Eric Badger commented on YARN-10493: [~MatthewSharp], thanks for the PR. Just starting to take a look at this now. I am wondering if the document is still up to date though. Is the PR you put up still a good reflection of what's in the document? Just want to make sure > RunC container repository v2 > > > Key: YARN-10493 > URL: https://issues.apache.org/jira/browse/YARN-10493 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, yarn >Affects Versions: 3.3.0 >Reporter: Craig Condit >Assignee: Matthew Sharp >Priority: Major > Labels: pull-request-available > Attachments: runc-container-repository-v2-design.pdf > > Time Spent: 0.5h > Remaining Estimate: 0h > > The current runc container repository design has scalability and usability > issues which will likely limit widespread adoption. We should address this > with a new, V2 layout. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10493) RunC container repository v2
[
https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305211#comment-17305211
]
Hadoop QA commented on YARN-10493:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
1s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue} 0m
1s{color} | | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 2 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 14m
5s{color} | | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
10s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
40s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
7s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
2s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
53s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
26s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
34s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 6m
43s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 32s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
25s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
37s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
37s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
17s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
17s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 52s{color} |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2789/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt]
| {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 2 new +
207 unchanged - 0 fixed = 209 total (was 207) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
32s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
11s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
32s{color} | | {color:green} the patch
[jira] [Commented] (YARN-10493) RunC container repository v2
[
https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305091#comment-17305091
]
Hadoop QA commented on YARN-10493:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
55s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue} 0m
1s{color} | | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 2 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 14m
23s{color} | | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
11s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
7s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
53s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
44s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
54s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 3m
35s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 50s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
28s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
28s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
46s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
46s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 39s{color} |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2789/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt]
| {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 7 new +
207 unchanged - 0 fixed = 214 total (was 207) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
44s{color} |
[jira] [Commented] (YARN-10493) RunC container repository v2
[ https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304979#comment-17304979 ] Matthew Sharp commented on YARN-10493: -- I have an initial PR to address the improvements outlined in the attached pdf. I have some thoughts around the manifest caching that I would like to address in a follow up Jira. We have this running internally with the Java CLI tool from YARN-10494. > RunC container repository v2 > > > Key: YARN-10493 > URL: https://issues.apache.org/jira/browse/YARN-10493 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, yarn >Affects Versions: 3.3.0 >Reporter: Craig Condit >Assignee: Matthew Sharp >Priority: Major > Labels: pull-request-available > Attachments: runc-container-repository-v2-design.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > The current runc container repository design has scalability and usability > issues which will likely limit widespread adoption. We should address this > with a new, V2 layout. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
