[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904182#comment-16904182
]
Eric Yang commented on YARN-9564:
---------------------------------
[~ebadger] Thank for the patch 003. The script seems to progress further with
some errors:
{code:bash}
$ ./docker_to_squash.py --hadoop-prefix=/usr/local/hadoop-3.3.0-SNAPSHOT
--working-dir=/tmp/sq --log=DEBUG pull-build-push-update
centos:latest,centos:latest
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'version']
DEBUG: command: ['skopeo', '-v']
DEBUG: command: ['mksquashfs', '-version']
DEBUG: args: Namespace(LOG_LEVEL='DEBUG', check_magic_file=False, force=False,
func=<function pull_build_push_update at 0x165ea28>,
hadoop_prefix='/usr/local/hadoop-3.3.0-SNAPSHOT', hdfs_root='/runc-root',
image_tag_to_hash='image-tag-to-hash',
images_and_tags=['centos:latest,centos:latest'],
magic_file='etc/dockerfile-version', pull_format='docker', replication=1,
skopeo_format='dir', sub_command='pull-build-push-update',
working_dir='/tmp/sq')
DEBUG: extra: []
DEBUG: image-tag-to-hash: image-tag-to-hash
DEBUG: LOG_LEVEL: DEBUG
DEBUG: HADOOP_BIN_DIR: /usr/local/hadoop-3.3.0-SNAPSHOT/bin
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root']
ls: `/runc-root': No such file or directory
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-mkdir',
'/runc-root']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root']
DEBUG: Setting up squashfs dirs: ['/runc-root/layers', '/runc-root/config',
'/runc-root/manifests']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/layers']
ls: `/runc-root/layers': No such file or directory
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-mkdir',
'/runc-root/layers']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/layers']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root/layers']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/config']
ls: `/runc-root/config': No such file or directory
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-mkdir',
'/runc-root/config']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/config']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root/config']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/manifests']
ls: `/runc-root/manifests': No such file or directory
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-mkdir',
'/runc-root/manifests']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/manifests']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root/manifests']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/image-tag-to-hash']
ls: `/runc-root/image-tag-to-hash': No such file or directory
INFO: Working on image centos:latest with tags ['centos:latest']
DEBUG: command: ['skopeo', 'inspect', '--raw', 'docker://centos:latest']
FATA[0000] error getting username and password: error reading JSON file
"/run/user/0/containers/auth.json": open /run/user/0/containers/auth.json:
permission denied
DEBUG: command: ['sudo', 'rm', '-rf', '/tmp/sq']
Traceback (most recent call last):
File "./docker_to_squash.py", line 1283, in <module>
main()
File "./docker_to_squash.py", line 1280, in main
args.func(args)
File "./docker_to_squash.py", line 560, in pull_build_push_update
manifest, manifest_hash = get_manifest_from_docker_image(pull_format, image)
File "./docker_to_squash.py", line 237, in get_manifest_from_docker_image
False, True, True)
File "./docker_to_squash.py", line 67, in shell_command
+ str(out) + "\nstderr: " + str(err))
Exception: Commmand: ['skopeo', 'inspect', '--raw', 'docker://centos:latest']
failed with returncode: 1
stdout:
stderr: None
{code}
It appears some commands are not expected to prompt password during execution.
Any hint?
> Create docker-to-squash tool for image conversion
> -------------------------------------------------
>
> Key: YARN-9564
> URL: https://issues.apache.org/jira/browse/YARN-9564
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Eric Badger
> Assignee: Eric Badger
> Priority: Major
> Attachments: YARN-9564.001.patch, YARN-9564.002.patch,
> YARN-9564.003.patch
>
>
> The new runc runtime uses docker images that are converted into multiple
> squashfs images. Each layer of the docker image will get its own squashfs
> image. We need a tool to help automate the creation of these squashfs images
> when all we have is a docker image
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
