[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[ https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970382#comment-16970382 ] Jim Brennan commented on YARN-9564: --- Thanks for the updates [~ebadger]! I am +1 (non-binding) on patch 006. > Create docker-to-squash tool for image conversion > - > > Key: YARN-9564 > URL: https://issues.apache.org/jira/browse/YARN-9564 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9564.001.patch, YARN-9564.002.patch, > YARN-9564.003.patch, YARN-9564.004.patch, YARN-9564.005.patch, > YARN-9564.006.patch > > > The new runc runtime uses docker images that are converted into multiple > squashfs images. Each layer of the docker image will get its own squashfs > image. We need a tool to help automate the creation of these squashfs images > when all we have is a docker image -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969770#comment-16969770
]
Hadoop QA commented on YARN-9564:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
48s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
11s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} pylint {color} | {color:orange} 0m
7s{color} | {color:orange} The patch generated 124 new + 0 unchanged - 0 fixed
= 124 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
17s{color} | {color:green} hadoop-assemblies in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 21s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}206m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.webproxy.TestWebAppProxyServlet |
| | hadoop.yarn.server.webproxy.amfilter.TestAmFilter |
| |
hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities |
| |
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
|
| |
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
|
| | hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema
|
| |
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity |
| | hadoop.yarn.server.timelineservice.storage.TestTimelineWriterHBaseDown |
| | hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain
|
| | hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown |
| | hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps |
| |
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969708#comment-16969708
]
Hadoop QA commented on YARN-9564:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
2s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} pylint {color} | {color:orange} 0m
5s{color} | {color:orange} The patch generated 121 new + 0 unchanged - 0 fixed
= 121 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 44s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hadoop-assemblies in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 22s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.webproxy.TestWebAppProxyServlet |
| | hadoop.yarn.server.webproxy.amfilter.TestAmFilter |
| | hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain
|
| | hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown |
| |
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
|
| |
hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities |
| | hadoop.yarn.server.timelineservice.storage.TestTimelineWriterHBaseDown |
| |
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
|
| | hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun |
| | hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps |
| |
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity |
| |
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[ https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969634#comment-16969634 ] Eric Badger commented on YARN-9564: --- Patch 006 adds a clearer error message when the hdfs runc root has not already been created. > Create docker-to-squash tool for image conversion > - > > Key: YARN-9564 > URL: https://issues.apache.org/jira/browse/YARN-9564 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9564.001.patch, YARN-9564.002.patch, > YARN-9564.003.patch, YARN-9564.004.patch, YARN-9564.005.patch, > YARN-9564.006.patch > > > The new runc runtime uses docker images that are converted into multiple > squashfs images. Each layer of the docker image will get its own squashfs > image. We need a tool to help automate the creation of these squashfs images > when all we have is a docker image -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969605#comment-16969605
]
Eric Badger commented on YARN-9564:
---
Patch 005 adds dependency checking for {{tar}} and {{setfattr}}.
> Create docker-to-squash tool for image conversion
> -
>
> Key: YARN-9564
> URL: https://issues.apache.org/jira/browse/YARN-9564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9564.001.patch, YARN-9564.002.patch,
> YARN-9564.003.patch, YARN-9564.004.patch, YARN-9564.005.patch
>
>
> The new runc runtime uses docker images that are converted into multiple
> squashfs images. Each layer of the docker image will get its own squashfs
> image. We need a tool to help automate the creation of these squashfs images
> when all we have is a docker image
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[ https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965038#comment-16965038 ] Jim Brennan commented on YARN-9564: --- Based on my testing with the latest patches for YARN-9561, YARN-9562, and this patch, I am +1 (non-binding) on patch 004. I was able to use docker2squash.py to pull a docker image, squash it, and push the layers to my local hdfs. I was then able to run some test jobs using the runc container runtime. > Create docker-to-squash tool for image conversion > - > > Key: YARN-9564 > URL: https://issues.apache.org/jira/browse/YARN-9564 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9564.001.patch, YARN-9564.002.patch, > YARN-9564.003.patch, YARN-9564.004.patch > > > The new runc runtime uses docker images that are converted into multiple > squashfs images. Each layer of the docker image will get its own squashfs > image. We need a tool to help automate the creation of these squashfs images > when all we have is a docker image -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[ https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954992#comment-16954992 ] Eric Badger commented on YARN-9564: --- I have added this as a blocker for YARN-9561, as the most recent patch on YARN-9561 is based off of patch 004 in this JIRA. Together with YARN-9562 patch 009 and YARN-9561 patch 005 this creates a fully working RuncContainerRuntime. YARN-9884.004.patch YARN-9561.005.patch YARN-9562.009.patch YARN-9564.004.patch > Create docker-to-squash tool for image conversion > - > > Key: YARN-9564 > URL: https://issues.apache.org/jira/browse/YARN-9564 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9564.001.patch, YARN-9564.002.patch, > YARN-9564.003.patch, YARN-9564.004.patch > > > The new runc runtime uses docker images that are converted into multiple > squashfs images. Each layer of the docker image will get its own squashfs > image. We need a tool to help automate the creation of these squashfs images > when all we have is a docker image -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911944#comment-16911944
]
Hadoop QA commented on YARN-9564:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
57s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
8s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
12s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} pylint {color} | {color:orange} 0m
7s{color} | {color:orange} The patch generated 119 new + 0 unchanged - 0 fixed
= 119 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
17s{color} | {color:green} hadoop-assemblies in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}212m
40s{color} | {color:green} hadoop-yarn in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
59s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}325m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9564 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12978121/YARN-9564.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml pylint |
| uname | Linux 6743a8320eb5 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3
08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3a145e2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| pylint | v1.9.2 |
| pylint |
https://builds.apache.org/job/PreCommit-YARN-Build/24605/artifact/out/diff-patch-pylint.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24605/testReport/ |
| Max. process+thread count | 858 (vs. ulimit of 5500) |
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911777#comment-16911777
]
Eric Badger commented on YARN-9564:
---
To run on a local image, you'll need to specify the {{--pull-format}} on the
command line. It defaults to docker, which will send you to the docker
registry. If you want to use a local image, use docker-daemon instead. e.g.
{{--pull-format docker-daemon}}
I've uploaded patch 004 to add functionality for docker-daemon support since
there were a few parsing issues in patch 003.
{noformat}
./docker_to_squash.py --log=DEBUG --pull-format docker-daemon
pull-build-push-update hadoop/rhel7:current,hadoop/rhel7:current
{noformat}
I was able to successfully run the tool on a local image (hadoop/rhel7:current)
using the above command
> Create docker-to-squash tool for image conversion
> -
>
> Key: YARN-9564
> URL: https://issues.apache.org/jira/browse/YARN-9564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9564.001.patch, YARN-9564.002.patch,
> YARN-9564.003.patch, YARN-9564.004.patch
>
>
> The new runc runtime uses docker images that are converted into multiple
> squashfs images. Each layer of the docker image will get its own squashfs
> image. We need a tool to help automate the creation of these squashfs images
> when all we have is a docker image
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910839#comment-16910839
]
Eric Yang commented on YARN-9564:
-
[~ebadger] what is the right procedure for building image locally and upload to
hdfs?
Tried various approach, and none seems to work:
to push-update from local docker image to hdfs:
{code}
docker images
REPOSITORYTAG IMAGE ID
CREATED SIZE
runc-centos latest 467de23cbfb5
16 minutes ago 545MB
local/runc-centos latest 467de23cbfb5
16 minutes ago 545MB
{code}
Pushing image to hdfs result in errors:
{code}
# ./docker_to_squash.py --hadoop-prefix=/usr/local/hadoop-3.3.0-SNAPSHOT
--log=DEBUG pull-build-push-update
local/runc-centos:latest,local/runc-centos:latest
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'version']
DEBUG: command: ['skopeo', '-v']
DEBUG: command: ['mksquashfs', '-version']
DEBUG: args: Namespace(LOG_LEVEL='DEBUG', check_magic_file=False, force=False,
func=,
hadoop_prefix='/usr/local/hadoop-3.3.0-SNAPSHOT', hdfs_root='/runc-root',
image_tag_to_hash='image-tag-to-hash',
images_and_tags=['local/runc-centos:latest,local/runc-centos:latest'],
magic_file='etc/dockerfile-version', pull_format='docker', replication=1,
skopeo_format='dir', sub_command='pull-build-push-update',
working_dir='dts-work-dir')
DEBUG: extra: []
DEBUG: image-tag-to-hash: image-tag-to-hash
DEBUG: LOG_LEVEL: DEBUG
DEBUG: HADOOP_BIN_DIR: /usr/local/hadoop-3.3.0-SNAPSHOT/bin
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root']
DEBUG: Setting up squashfs dirs: ['/runc-root/layers', '/runc-root/config',
'/runc-root/manifests']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/layers']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root/layers']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/config']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root/config']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/manifests']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root/manifests']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/image-tag-to-hash']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-get',
'/runc-root/image-tag-to-hash', 'dts-work-dir/image-tag-to-hash']
2019-08-19 23:35:55,405 INFO sasl.SaslDataTransferClient: SASL encryption trust
check: localHostTrusted = false, remoteHostTrusted = false
INFO: Working on image local/runc-centos:latest with tags
['local/runc-centos:latest']
DEBUG: command: ['sudo', 'skopeo', 'inspect', '--raw',
'docker://local/runc-centos:latest']
FATA[0001] Error reading manifest latest in docker.io/local/runc-centos: errors:
denied: requested access to the resource is denied
unauthorized: authentication required
DEBUG: command: ['sudo', 'rm', '-rf', 'dts-work-dir']
Traceback (most recent call last):
File "./docker_to_squash.py", line 1283, in
main()
File "./docker_to_squash.py", line 1280, in main
args.func(args)
File "./docker_to_squash.py", line 560, in pull_build_push_update
manifest, manifest_hash = get_manifest_from_docker_image(pull_format, image)
File "./docker_to_squash.py", line 237, in get_manifest_from_docker_image
False, True, True)
File "./docker_to_squash.py", line 67, in shell_command
+ str(out) + "\nstderr: " + str(err))
Exception: Commmand: ['sudo', 'skopeo', 'inspect', '--raw',
'docker://local/runc-centos:latest'] failed with returncode: 1
stdout:
stderr: None
{code}
> Create docker-to-squash tool for image conversion
> -
>
> Key: YARN-9564
> URL: https://issues.apache.org/jira/browse/YARN-9564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9564.001.patch, YARN-9564.002.patch,
> YARN-9564.003.patch
>
>
> The new runc runtime uses docker images that are converted into multiple
> squashfs images. Each layer of the docker image will get its own squashfs
> image. We need a tool to help automate the creation of these squashfs images
> when all we have is a docker image
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail:
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[ https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905579#comment-16905579 ] Eric Badger commented on YARN-9564: --- [~eyang], those logs are normal. When debug logging is enabled, the output of all shell commands is logged. The script checks to see if files exist or not before downloading or uploading them. I'm not sure if it's better to suppress this logging regardless or debug logging or not. > Create docker-to-squash tool for image conversion > - > > Key: YARN-9564 > URL: https://issues.apache.org/jira/browse/YARN-9564 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9564.001.patch, YARN-9564.002.patch, > YARN-9564.003.patch > > > The new runc runtime uses docker images that are converted into multiple > squashfs images. Each layer of the docker image will get its own squashfs > image. We need a tool to help automate the creation of these squashfs images > when all we have is a docker image -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905554#comment-16905554
]
Eric Yang commented on YARN-9564:
-
[~ebadger] This script works with root user only and there are a number of
output that looks like this:
{code}ls:
`/runc-root/manifests/ca58fe458b8d94bc6e3072f1cfbd334855858e05e1fd633aa07cf7f82b048e66':
No such file or directory{code}
> Create docker-to-squash tool for image conversion
> -
>
> Key: YARN-9564
> URL: https://issues.apache.org/jira/browse/YARN-9564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9564.001.patch, YARN-9564.002.patch,
> YARN-9564.003.patch
>
>
> The new runc runtime uses docker images that are converted into multiple
> squashfs images. Each layer of the docker image will get its own squashfs
> image. We need a tool to help automate the creation of these squashfs images
> when all we have is a docker image
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904219#comment-16904219
]
Eric Badger commented on YARN-9564:
---
The skopeo error looks to be because {{XDG_RUNTIME_DIR}} is set to /run/user/0,
which is root-owned. If you unset this environment variable or set it to
/run/user/$UID, I believe skopeo should work.
The permission failure in your 2nd comment is a result of running skopeo as
root and then the script not being able to remove the root-owned file as a
normal user.
As for running the script as root, yes, I would like to remove root altogether.
However, in the meantime, you should be able to get around the issue by running
as root with a kerberos credential.
> Create docker-to-squash tool for image conversion
> -
>
> Key: YARN-9564
> URL: https://issues.apache.org/jira/browse/YARN-9564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9564.001.patch, YARN-9564.002.patch,
> YARN-9564.003.patch
>
>
> The new runc runtime uses docker images that are converted into multiple
> squashfs images. Each layer of the docker image will get its own squashfs
> image. We need a tool to help automate the creation of these squashfs images
> when all we have is a docker image
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904188#comment-16904188
]
Eric Yang commented on YARN-9564:
-
If I added sudo to skopeo, the script only fail on removing temp directory. I
think this is toward the end of conversion process. I think this script is
suffering from mixed identity that some operations must be done as root, and
some operation must be done as hdfs admin. If running this script as root user
with a kerberos credential to hdfs, the script is likely pass. However, most
admin will require to run this script has themselves to avoid getting root user
mixed up with normal user kerberos ticket cache. In the later case, the
current script would encounter permission denied errors.
{code}
DEBUG: command: ['sudo', 'mksquashfs',
u'/tmp/sq/expand_archive_8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df',
u'/tmp/sq/centos:latest/8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df.sqsh']
DEBUG: command: ['sudo', 'rm', '-rf', '/tmp/sq']
Traceback (most recent call last):
File "./docker_to_squash.py", line 1283, in
main()
File "./docker_to_squash.py", line 1280, in main
args.func(args)
File "./docker_to_squash.py", line 612, in pull_build_push_update
docker_to_squash(skopeo_dir, layer, working_dir)
File "./docker_to_squash.py", line 512, in docker_to_squash
os.remove(layer_path)
OSError: [Errno 13] Permission denied:
'/tmp/sq/centos:latest/8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df'
{code}
> Create docker-to-squash tool for image conversion
> -
>
> Key: YARN-9564
> URL: https://issues.apache.org/jira/browse/YARN-9564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9564.001.patch, YARN-9564.002.patch,
> YARN-9564.003.patch
>
>
> The new runc runtime uses docker images that are converted into multiple
> squashfs images. Each layer of the docker image will get its own squashfs
> image. We need a tool to help automate the creation of these squashfs images
> when all we have is a docker image
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904182#comment-16904182
]
Eric Yang commented on YARN-9564:
-
[~ebadger] Thank for the patch 003. The script seems to progress further with
some errors:
{code:bash}
$ ./docker_to_squash.py --hadoop-prefix=/usr/local/hadoop-3.3.0-SNAPSHOT
--working-dir=/tmp/sq --log=DEBUG pull-build-push-update
centos:latest,centos:latest
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'version']
DEBUG: command: ['skopeo', '-v']
DEBUG: command: ['mksquashfs', '-version']
DEBUG: args: Namespace(LOG_LEVEL='DEBUG', check_magic_file=False, force=False,
func=,
hadoop_prefix='/usr/local/hadoop-3.3.0-SNAPSHOT', hdfs_root='/runc-root',
image_tag_to_hash='image-tag-to-hash',
images_and_tags=['centos:latest,centos:latest'],
magic_file='etc/dockerfile-version', pull_format='docker', replication=1,
skopeo_format='dir', sub_command='pull-build-push-update',
working_dir='/tmp/sq')
DEBUG: extra: []
DEBUG: image-tag-to-hash: image-tag-to-hash
DEBUG: LOG_LEVEL: DEBUG
DEBUG: HADOOP_BIN_DIR: /usr/local/hadoop-3.3.0-SNAPSHOT/bin
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root']
ls: `/runc-root': No such file or directory
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-mkdir',
'/runc-root']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root']
DEBUG: Setting up squashfs dirs: ['/runc-root/layers', '/runc-root/config',
'/runc-root/manifests']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/layers']
ls: `/runc-root/layers': No such file or directory
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-mkdir',
'/runc-root/layers']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/layers']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root/layers']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/config']
ls: `/runc-root/config': No such file or directory
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-mkdir',
'/runc-root/config']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/config']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root/config']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/manifests']
ls: `/runc-root/manifests': No such file or directory
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-mkdir',
'/runc-root/manifests']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/manifests']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root/manifests']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/image-tag-to-hash']
ls: `/runc-root/image-tag-to-hash': No such file or directory
INFO: Working on image centos:latest with tags ['centos:latest']
DEBUG: command: ['skopeo', 'inspect', '--raw', 'docker://centos:latest']
FATA[] error getting username and password: error reading JSON file
"/run/user/0/containers/auth.json": open /run/user/0/containers/auth.json:
permission denied
DEBUG: command: ['sudo', 'rm', '-rf', '/tmp/sq']
Traceback (most recent call last):
File "./docker_to_squash.py", line 1283, in
main()
File "./docker_to_squash.py", line 1280, in main
args.func(args)
File "./docker_to_squash.py", line 560, in pull_build_push_update
manifest, manifest_hash = get_manifest_from_docker_image(pull_format, image)
File "./docker_to_squash.py", line 237, in get_manifest_from_docker_image
False, True, True)
File "./docker_to_squash.py", line 67, in shell_command
+ str(out) + "\nstderr: " + str(err))
Exception: Commmand: ['skopeo', 'inspect', '--raw', 'docker://centos:latest']
failed with returncode: 1
stdout:
stderr: None
{code}
It appears some commands are not expected to prompt password during execution.
Any hint?
> Create docker-to-squash tool for image conversion
> -
>
> Key: YARN-9564
> URL: https://issues.apache.org/jira/browse/YARN-9564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9564.001.patch, YARN-9564.002.patch,
> YARN-9564.003.patch
>
>
> The new runc runtime uses docker images that are converted into multiple
> squashfs images. Each layer of the docker image will get its own
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903520#comment-16903520
]
Hadoop QA commented on YARN-9564:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 15s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} pylint {color} | {color:orange} 0m
5s{color} | {color:orange} The patch generated 132 new + 0 unchanged - 0 fixed
= 132 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
15s{color} | {color:green} hadoop-assemblies in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}145m 46s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
55s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}247m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.timelineservice.storage.TestTimelineWriterHBaseDown |
| | hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9564 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977083/YARN-9564.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml pylint |
| uname | Linux 9bd93ec5b9d2 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3
08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / aa5f445 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| pylint | v1.9.2 |
| pylint |
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903391#comment-16903391
]
Eric Badger commented on YARN-9564:
---
{noformat}
[ebadger@foobar sbin]$ pwd
/home/ebadger/hadoop/hadoop-dist/target/hadoop-3.3.0-SNAPSHOT/sbin
[ebadger@foobar sbin]$ ls
distribute-exclude.sh hadoop-daemons.shrefresh-namenodes.sh
start-dfs.cmdstart-yarn.sh stop-dfs.cmdstop-yarn.sh
docker_to_squash.pyhttpfs.shstart-all.cmd
start-dfs.sh stop-all.cmd stop-dfs.sh workers.sh
FederationStateStore kms.sh start-all.sh
start-secure-dns.sh stop-all.sh stop-secure-dns.sh yarn-daemon.sh
hadoop-daemon.sh mr-jobhistory-daemon.sh start-balancer.sh
start-yarn.cmd stop-balancer.sh stop-yarn.cmd yarn-daemons.sh
[ebadger@foobar sbin]$ hadoop fs -ls /
Found 3 items
drwxrwx--- - ebadger supergroup 0 2019-08-07 19:35 /home
drwx-- - ebadger supergroup 0 2019-08-07 19:35 /tmp
drwx-- - ebadger supergroup 0 2019-08-07 19:35 /user
[ebadger@foobar sbin]$ ./docker_to_squash.py --working-dir /tmp --log=DEBUG
pull-build-push-update centos:latest,centos
DEBUG: command: ['/hadoop-2.8.6-SNAPSHOT/bin/hadoop', 'version']
DEBUG: command: ['skopeo', '-v']
DEBUG: command: ['mksquashfs', '-version']
DEBUG: args: Namespace(LOG_LEVEL='DEBUG', check_magic_file=False, force=False,
func=,
hadoop_prefix='/hadoop-2.8.6-SNAPSHOT', hdfs_root='/runc-root',
image_tag_to_hash='image-tag-to-hash',
images_and_tags=['centos:latest,centos'], magic_file='etc/dockerfile-version',
pull_format='docker', replication=1, skopeo_format='dir',
sub_command='pull-build-push-update', working_dir='/tmp')
DEBUG: extra: []
DEBUG: image-tag-to-hash: image-tag-to-hash
DEBUG: LOG_LEVEL: DEBUG
DEBUG: HADOOP_BIN_DIR: /hadoop-2.8.6-SNAPSHOT/bin
DEBUG: command: ['/hadoop-2.8.6-SNAPSHOT/bin/hadoop', 'fs', '-ls', '/runc-root']
ls: `/runc-root': No such file or directory
DEBUG: command: ['/hadoop-2.8.6-SNAPSHOT/bin/hadoop', 'fs', '-mkdir',
'/runc-root']
DEBUG: command: ['/hadoop-2.8.6-SNAPSHOT/bin/hadoop', 'fs', '-ls', '/runc-root']
DEBUG: command: ['/hadoop-2.8.6-SNAPSHOT/bin/hadoop', 'fs', '-chmod', '755',
'/runc-root']
DEBUG: Setting up squashfs dirs: ['/runc-root/layers', '/runc-root/config',
'/runc-root/manifests']
DEBUG: command: ['/hadoop-2.8.6-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/layers']
ls: `/runc-root/layers': No such file or directory
DEBUG: command: ['/hadoop-2.8.6-SNAPSHOT/bin/hadoop', 'fs', '-mkdir',
'/runc-root/layers']
DEBUG: command: ['/hadoop-2.8.6-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/layers']
DEBUG: command: ['/hadoop-2.8.6-SNAPSHOT/bin/hadoop', 'fs', '-chmod', '755',
'/runc-root/layers']
DEBUG: command: ['/hadoop-2.8.6-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/config']
ls: `/runc-root/config': No such file or directory
DEBUG: command: ['/hadoop-2.8.6-SNAPSHOT/bin/hadoop', 'fs', '-mkdir',
'/runc-root/config']
DEBUG: command: ['/hadoop-2.8.6-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/config']
DEBUG: command: ['/hadoop-2.8.6-SNAPSHOT/bin/hadoop', 'fs', '-chmod', '755',
'/runc-root/config']
DEBUG: command: ['/hadoop-2.8.6-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/manifests']
ls: `/runc-root/manifests': No such file or directory
DEBUG: command: ['/hadoop-2.8.6-SNAPSHOT/bin/hadoop', 'fs', '-mkdir',
'/runc-root/manifests']
DEBUG: command: ['/hadoop-2.8.6-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/manifests']
DEBUG: command: ['/hadoop-2.8.6-SNAPSHOT/bin/hadoop', 'fs', '-chmod', '755',
'/runc-root/manifests']
DEBUG: command: ['/hadoop-2.8.6-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/image-tag-to-hash']
ls: `/runc-root/image-tag-to-hash': No such file or directory
INFO: Working on image centos:latest with tags ['centos']
DEBUG: command: ['skopeo', 'inspect', '--raw', 'docker://centos:latest']
DEBUG: skopeo inspect --raw returned a list of manifests
DEBUG: amd64 manifest sha is:
sha256:ca58fe458b8d94bc6e3072f1cfbd334855858e05e1fd633aa07cf7f82b048e66
DEBUG: command: ['skopeo', 'inspect', '--raw',
u'docker://centos@sha256:ca58fe458b8d94bc6e3072f1cfbd334855858e05e1fd633aa07cf7f82b048e66']
INFO: manifest: {u'layers': [{u'mediaType':
u'application/vnd.docker.image.rootfs.diff.tar.gzip', u'digest':
u'sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df',
u'size': 75403831}], u'schemaVersion': 2, u'config': {u'mediaType':
u'application/vnd.docker.container.image.v1+json', u'digest':
u'sha256:9f38484d220fa527b1fb19747638497179500a1bed8bf0498eb788229229e6e1',
u'size': 2182}, u'mediaType':
u'application/vnd.docker.distribution.manifest.v2+json'}
INFO: manifest: {u'layers': [{u'mediaType':
u'application/vnd.docker.image.rootfs.diff.tar.gzip', u'digest':
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901604#comment-16901604
]
Hadoop QA commented on YARN-9564:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} pylint {color} | {color:orange} 0m
6s{color} | {color:orange} The patch generated 113 new + 0 unchanged - 0 fixed
= 113 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} hadoop-assemblies in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}203m
15s{color} | {color:green} hadoop-yarn in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
48s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}296m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9564 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12976847/YARN-9564.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml pylint |
| uname | Linux b5a644c5e0af 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b77761b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| pylint | v1.9.2 |
| pylint |
https://builds.apache.org/job/PreCommit-YARN-Build/24483/artifact/out/diff-patch-pylint.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24483/testReport/ |
| Max. process+thread count | 896 (vs. ulimit of 1) |
|
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901500#comment-16901500
]
Eric Yang commented on YARN-9564:
-
[~ebadger] {quote}Yes, you'll need to run this script as sudo due to a few of
the commands in the script. It's probably easiest to run the whole script as
root, but I like to run as little as possible as root. I could require the
script run as root and then drop privileges when they aren't needed.{quote}
My first choice is to route this operation via YARN daemon to perform the
privilege escalation operations similar to container launch. This would put
the usage of the command similar to "docker build" that requires a trusted
daemon to validate security then perform the image build operation. The second
choice is to check the current user is a privileged user and run accordingly.
{quote}I believe that Craig Condit has a java version of something similar to
this tool. I don't think that I'm going to have time to rewrite this in Java,
but we might be able to leverage his tool if you think that approach is
better.{quote}
[~ccondit]'s tool provides basic flatten of docker image to squashfs. The
python implementation provides more metadata management of layers. Unless
there is effort to add metadata management to Craig's version. They are not
equal in functionality. I can not recommend to change direction unless someone
willing put in the effort to covert the python work to Java. I am shy from
committing python version at this time because external dependencies make this
script incomplete to function as standalone unit. I think it is too hard to
replicate for normal users. The script can improves on detecting
pre-requisites dependencies ahead of time instead of error out half way of the
script execution.
> Create docker-to-squash tool for image conversion
> -
>
> Key: YARN-9564
> URL: https://issues.apache.org/jira/browse/YARN-9564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9564.001.patch, YARN-9564.002.patch
>
>
> The new runc runtime uses docker images that are converted into multiple
> squashfs images. Each layer of the docker image will get its own squashfs
> image. We need a tool to help automate the creation of these squashfs images
> when all we have is a docker image
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[ https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901413#comment-16901413 ] Eric Badger commented on YARN-9564: --- Patch 002 addresses the rest of your review comments, [~eyang]. I still have open questions from your comments #5 and #7 from my previous comment. > Create docker-to-squash tool for image conversion > - > > Key: YARN-9564 > URL: https://issues.apache.org/jira/browse/YARN-9564 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9564.001.patch, YARN-9564.002.patch > > > The new runc runtime uses docker images that are converted into multiple > squashfs images. Each layer of the docker image will get its own squashfs > image. We need a tool to help automate the creation of these squashfs images > when all we have is a docker image -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[ https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900209#comment-16900209 ] Eric Badger commented on YARN-9564: --- Hi [~eyang], appreciate the comments. First and foremost, were you eventually able to get the script to work and correctly setup the squashFS layers, config, manifest, and image-tag-to-hash file? I know that the script is far from perfect and certainly has bugs or improvements to make. But my main reason for posting it was to help setup the runC environment so that you can get YARN-9562 and YARN-9561 running. bq. 5. This script requires password less sudo. This is usually a no-no from Enterprise user. What can we do about these sudo commands? Maybe assume this script require admin privileges? Yes, you'll need to run this script as sudo due to a few of the commands in the script. It's probably easiest to run the whole script as root, but I like to run as little as possible as root. I could require the script run as root and then drop privileges when they aren't needed. bq. 7. How about having this writing in Java? I believe that [~ccondit] has a java version of something similar to this tool. I don't think that I'm going to have time to rewrite this in Java, but we might be able to leverage his tool if you think that approach is better. I will fix the rest of the issues that you raised. > Create docker-to-squash tool for image conversion > - > > Key: YARN-9564 > URL: https://issues.apache.org/jira/browse/YARN-9564 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9564.001.patch > > > The new runc runtime uses docker images that are converted into multiple > squashfs images. Each layer of the docker image will get its own squashfs > image. We need a tool to help automate the creation of these squashfs images > when all we have is a docker image -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[ https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899283#comment-16899283 ] Eric Yang commented on YARN-9564: - [~ebadger] Thank you for the patch. # I think this is a user facing script, the source location for user facing script resides in hadoop-yarn/bin. It maybe better to relocate the script to this location. I am also concerned that there is no python user facing script in Hadoop today. This script does not follow the Hadoop command line practice. I am not sure if it is possible to create a shell wrapper on yarn cli to invoke this script from libexec to make this script less stand out, but I let others weight in on usability aspect of this script. # If subprocess raised exception, process.kill() should be called to ensure there is no orphan child process hanging around. # The script accepts hadoop_prefix, but the hadoop shell commands are using binary on $PATH variable. This might not work out of the box. Hadoop shell script may help the bootstrap process by creating hadoop environment variables to work with the python script to provide a better user experience for Hadoop deployment that doesn't rely on PATH variable. # Not sure how to work with working_dir parameter, it seems to fail if /tmp is specified. Unless user give /tmp/test and test must not exist. # This script requires password less sudo. This is usually a no-no from Enterprise user. What can we do about these sudo commands? Maybe assume this script require admin privileges? # Missing Apache license header. # How about having this writing in Java? # get_local_manifest_from_path didn't close file descriptor. # This utility depends on skopeo and python libraries. We may need a way to manage python dependencies, if no plan on rewriting this in Java. At minimum document the dependency package and version number required to run this script as comments in the script. > Create docker-to-squash tool for image conversion > - > > Key: YARN-9564 > URL: https://issues.apache.org/jira/browse/YARN-9564 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9564.001.patch > > > The new runc runtime uses docker images that are converted into multiple > squashfs images. Each layer of the docker image will get its own squashfs > image. We need a tool to help automate the creation of these squashfs images > when all we have is a docker image -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898398#comment-16898398
]
Hadoop QA commented on YARN-9564:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} pylint {color} | {color:orange} 0m
8s{color} | {color:orange} The patch generated 1090 new + 0 unchanged - 0 fixed
= 1090 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-assemblies in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m
51s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
42s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}101m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9564 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12976467/YARN-9564.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml pylint |
| uname | Linux f24d1fdcd0ae 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e111789 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| pylint | v1.9.2 |
| pylint |
https://builds.apache.org/job/PreCommit-YARN-Build/24445/artifact/out/diff-patch-pylint.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24445/testReport/ |
| asflicense |
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898345#comment-16898345
]
Eric Badger commented on YARN-9564:
---
Once the tool completes, your hdfs-root should look something like this. There
may be more or less layers depending on the image that you choose to build.
{noformat}
[ebadger@foo bin]$ hadoop fs -ls /runc-root/*
Found 1 items
-r--r--r-- 1 ebadger supergroup 8038 2019-08-01 20:22
/runc-root/config/599579330b2d6f01817b402be0cdc0af1a9b7b076f3743483c2c509b7f73c5ad
-r--r--r-- 1 ebadger supergroup153 2019-08-01 20:22
/runc-root/image-tag-to-hash
Found 10 items
-r--r--r-- 1 ebadger supergroup 1143881728 2019-08-01 20:19
/runc-root/layers/1b5ed3a410f4c2e8948816ceaae20aa2f2ba9aab682ca899f900fc4d43f6e98b.sqsh
-r--r--r-- 1 ebadger supergroup 4096 2019-08-01 20:21
/runc-root/layers/23c1e1af124ca00519aa2858e891a4f8c3a25e9350b39419b786b638f312f8d6.sqsh
-r--r--r-- 1 ebadger supergroup 4096 2019-08-01 20:20
/runc-root/layers/253aace22c99909557f14eebe73d5a56b3c155cc3ce1a941755d53fe10d53177.sqsh
-r--r--r-- 1 ebadger supergroup 106979328 2019-08-01 20:20
/runc-root/layers/50ede125b97f8bc1c4637b17c5d5fb93cd82088ad07d8143e07482cfbe025f6e.sqsh
-r--r--r-- 1 ebadger supergroup 4096 2019-08-01 20:22
/runc-root/layers/8227f11724713c930b1eed248657e725d200bf58cb2796cd1d2ed6404b49a5a9.sqsh
-r--r--r-- 1 ebadger supergroup1748992 2019-08-01 20:19
/runc-root/layers/ae4a0c3146659d5a83077c7143eb255c5115223ef346eb462124c9fe637d288a.sqsh
-r--r--r-- 1 ebadger supergroup 28672 2019-08-01 20:19
/runc-root/layers/b38724009ad1954978bb8c0943d6463bab47c7aab3f463085f7c0ba5fcf30145.sqsh
-r--r--r-- 1 ebadger supergroup 360402944 2019-08-01 20:20
/runc-root/layers/db1a08a14163561a7c7b78ecdc8a98211530b50a1a3eed9483bd556f688788f9.sqsh
-r--r--r-- 1 ebadger supergroup 4096 2019-08-01 20:21
/runc-root/layers/eb8f6b81108d2f3f5dd1de79805a34be5561ebb247aa8d4f87f096a9109335b9.sqsh
-r--r--r-- 1 ebadger supergroup 114610176 2019-08-01 20:21
/runc-root/layers/fb4e5c73119d8f54a37f12055ab9aa9d12e1e77c015742c86175a6a6d8b6e5fb.sqsh
Found 1 items
-r--r--r-- 1 ebadger supergroup 2418 2019-08-01 20:22
/runc-root/manifests/b71721ecf1e6cbbae071621b22f9003f6909b24bd883d804fef61ec2548bfda9
{noformat}
Your image-tag-to-hash file should look something like this:
{noformat}
[ebadger@foo bin]$ hadoop fs -cat /runc-root/image-tag-to-hash
busybox:latest:b71721ecf1e6cbbae071621b22f9003f6909b24bd883d804fef61ec2548bfda9#registry.hub.docker.com/library/busybox,busybox:latest
{noformat}
Note that the hashes used above do not actually reflect the exact hashes that
busybox will get you.
> Create docker-to-squash tool for image conversion
> -
>
> Key: YARN-9564
> URL: https://issues.apache.org/jira/browse/YARN-9564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9564.001.patch
>
>
> The new runc runtime uses docker images that are converted into multiple
> squashfs images. Each layer of the docker image will get its own squashfs
> image. We need a tool to help automate the creation of these squashfs images
> when all we have is a docker image
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898342#comment-16898342
]
Eric Badger commented on YARN-9564:
---
{noformat}
[ebadger@foo bin]$ ./docker-to-squash.py -h
usage: docker-to-squash.py [-h] [--working-dir WORKING_DIR]
[--skopeo-format SKOPEO_FORMAT]
[--pull-format PULL_FORMAT] [-l LOG_LEVEL]
[--hdfs-root HDFS_ROOT]
[--image-tag-to-hash IMAGE_TAG_TO_HASH]
[-r REPLICATION] [--hadoop-prefix HADOOP_PREFIX]
[-f] [--check-magic-file] [--magic-file MAGIC_FILE]
{pull-build-push-update,pull-build,push-update,remove-image,remove-tag,add-tag,copy-update,query-tag,list-tags}
...
positional arguments:
{pull-build-push-update,pull-build,push-update,remove-image,remove-tag,add-tag,copy-update,query-tag,list-tags}
sub help
pull-build-push-update
Pull an image, build its squashfs layers, push it to
hdfs, and atomically update the image-tag-to-hash file
pull-build Pull an image and build its squashfs layers
push-update Push the squashfs layers to hdfs and update the image-
tag-to-hash file
remove-imageRemove an image (manifest, config, layers) from hdfs
based on its tag or manifest hash
remove-tag Remove an image to tag mapping in the image-tag-to-
hash file
add-tag Add an image to tag mapping in the image-tag-to-hash
file
copy-update Copy an image from hdfs in one cluster to another and
then update the image-tag-to-hash file
query-tag Get the manifest, config, and layers associated with a
tag
list-tags List all tags in image-tag-to-hash file
optional arguments:
-h, --helpshow this help message and exit
--working-dir WORKING_DIR
Name of working directory
--skopeo-format SKOPEO_FORMAT
Output format for skopeo copy
--pull-format PULL_FORMAT
Pull format for skopeo
-l LOG_LEVEL, --log LOG_LEVEL
Logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
--hdfs-root HDFS_ROOT
The root directory in HDFS for all of the squashfs
images
--image-tag-to-hash IMAGE_TAG_TO_HASH
image-tag-to-hash filepath or filename in hdfs
-r REPLICATION, --replication REPLICATION
Replication factor for all files uploaded to HDFS
--hadoop-prefix HADOOP_PREFIX
hadoop_prefix value for environment
-f, --force Force overwrites in HDFS
--check-magic-fileCheck for a specific magic file in the image before
uploading
--magic-file MAGIC_FILE
The magic file to check for in the image
{noformat}
{noformat:Building and pushing a new image to HDFS}
./docker-to-squash.py --log=DEBUG pull-build-push-update ,
{noformat}
{noformat:title=Example}
./docker-to-squash.py --log=DEBUG pull-build-push-update
registry.hub.docker.com/library/busybox,busybox:latest
{noformat}
Note that the busybox image won't be enough to run the runC containers, since
it won't have Java, some native libs, and any other Hadoop things that are
needed to start a container. So you should point this to a Docker image that
you can run DockerLinuxContainerRuntime with.
> Create docker-to-squash tool for image conversion
> -
>
> Key: YARN-9564
> URL: https://issues.apache.org/jira/browse/YARN-9564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9564.001.patch
>
>
> The new runc runtime uses docker images that are converted into multiple
> squashfs images. Each layer of the docker image will get its own squashfs
> image. We need a tool to help automate the creation of these squashfs images
> when all we have is a docker image
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
