[ https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904182#comment-16904182 ]
Eric Yang commented on YARN-9564: --------------------------------- [~ebadger] Thank for the patch 003. The script seems to progress further with some errors: {code:bash} $ ./docker_to_squash.py --hadoop-prefix=/usr/local/hadoop-3.3.0-SNAPSHOT --working-dir=/tmp/sq --log=DEBUG pull-build-push-update centos:latest,centos:latest DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'version'] DEBUG: command: ['skopeo', '-v'] DEBUG: command: ['mksquashfs', '-version'] DEBUG: args: Namespace(LOG_LEVEL='DEBUG', check_magic_file=False, force=False, func=<function pull_build_push_update at 0x165ea28>, hadoop_prefix='/usr/local/hadoop-3.3.0-SNAPSHOT', hdfs_root='/runc-root', image_tag_to_hash='image-tag-to-hash', images_and_tags=['centos:latest,centos:latest'], magic_file='etc/dockerfile-version', pull_format='docker', replication=1, skopeo_format='dir', sub_command='pull-build-push-update', working_dir='/tmp/sq') DEBUG: extra: [] DEBUG: image-tag-to-hash: image-tag-to-hash DEBUG: LOG_LEVEL: DEBUG DEBUG: HADOOP_BIN_DIR: /usr/local/hadoop-3.3.0-SNAPSHOT/bin DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls', '/runc-root'] ls: `/runc-root': No such file or directory DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-mkdir', '/runc-root'] DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls', '/runc-root'] DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod', '755', '/runc-root'] DEBUG: Setting up squashfs dirs: ['/runc-root/layers', '/runc-root/config', '/runc-root/manifests'] DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls', '/runc-root/layers'] ls: `/runc-root/layers': No such file or directory DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-mkdir', '/runc-root/layers'] DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls', '/runc-root/layers'] DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod', '755', '/runc-root/layers'] DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls', '/runc-root/config'] ls: `/runc-root/config': No such file or directory DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-mkdir', '/runc-root/config'] DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls', '/runc-root/config'] DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod', '755', '/runc-root/config'] DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls', '/runc-root/manifests'] ls: `/runc-root/manifests': No such file or directory DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-mkdir', '/runc-root/manifests'] DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls', '/runc-root/manifests'] DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod', '755', '/runc-root/manifests'] DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls', '/runc-root/image-tag-to-hash'] ls: `/runc-root/image-tag-to-hash': No such file or directory INFO: Working on image centos:latest with tags ['centos:latest'] DEBUG: command: ['skopeo', 'inspect', '--raw', 'docker://centos:latest'] FATA[0000] error getting username and password: error reading JSON file "/run/user/0/containers/auth.json": open /run/user/0/containers/auth.json: permission denied DEBUG: command: ['sudo', 'rm', '-rf', '/tmp/sq'] Traceback (most recent call last): File "./docker_to_squash.py", line 1283, in <module> main() File "./docker_to_squash.py", line 1280, in main args.func(args) File "./docker_to_squash.py", line 560, in pull_build_push_update manifest, manifest_hash = get_manifest_from_docker_image(pull_format, image) File "./docker_to_squash.py", line 237, in get_manifest_from_docker_image False, True, True) File "./docker_to_squash.py", line 67, in shell_command + str(out) + "\nstderr: " + str(err)) Exception: Commmand: ['skopeo', 'inspect', '--raw', 'docker://centos:latest'] failed with returncode: 1 stdout: stderr: None {code} It appears some commands are not expected to prompt password during execution. Any hint? > Create docker-to-squash tool for image conversion > ------------------------------------------------- > > Key: YARN-9564 > URL: https://issues.apache.org/jira/browse/YARN-9564 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Eric Badger > Assignee: Eric Badger > Priority: Major > Attachments: YARN-9564.001.patch, YARN-9564.002.patch, > YARN-9564.003.patch > > > The new runc runtime uses docker images that are converted into multiple > squashfs images. Each layer of the docker image will get its own squashfs > image. We need a tool to help automate the creation of these squashfs images > when all we have is a docker image -- This message was sent by Atlassian JIRA (v7.6.14#76016) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org