[
https://issues.apache.org/jira/browse/YARN-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307481#comment-17307481
]
Eric Badger commented on YARN-10493:
------------------------------------
Additionally, I've run into some issues while testing.
{noformat:title=CLI Invocation}
hadoop jar ./hadoop-tools/hadoop-runc/target/hadoop-runc-3.4.0-SNAPSHOT.jar
org.apache.hadoop.runc.tools.ImportDockerImage -r docker.foobar.com:9999
hadoop-images/hadoop/rhel7 hadoop/rhel7
{noformat}
{noformat}
[ebadger@foo hadoop]$ hadoop fs -ls
/runc-root/meta/hadoop/rhel7@latest.properties
-rw------- 10 ebadger supergroup 236 2021-03-24 00:15
/runc-root/meta/hadoop/rhel7@latest.properties
{noformat}
Here's the properties file after the CLI tool completes.
{noformat}
<property>
<name>yarn.nodemanager.runtime.linux.runc.image-tag-to-manifest-plugin</name>
<value>org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.runc.ImageTagToManifestV2Plugin</value>
</property>
<property>
<name>yarn.nodemanager.runtime.linux.runc.manifest-to-resources-plugin</name>
<value>org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.runc.HdfsManifestToResourcesV2Plugin</value>
</property>
{noformat}
Then I set these properties as well as adding {{runc}} to the allowed-runtimes
config.
{noformat}
export
vars="YARN_CONTAINER_RUNTIME_TYPE=runc,YARN_CONTAINER_RUNTIME_RUNC_IMAGE=hadoop/rhel7";
$HADOOP_HOME/bin/hadoop jar
$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.*-tests.jar
sleep -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME=$HADOOP_HOME"
-Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME=$HADOOP_HOME"
-Dyarn.app.mapreduce.am.env=$vars -Dmapreduce.map.env=$vars
-Dmapreduce.reduce.env=$vars -mt 1 -rt 1 -m 1 -r 1
{noformat}
I ran a sleep job using this command.
{noformat}
2021-03-24 00:26:07,823 DEBUG [NM ContainerManager dispatcher]
runc.ImageTagToManifestV2Plugin
(ImageTagToManifestV2Plugin.java:getHdfsImageToHashReader(144)) - Checking HDFS
for image file: /runc-root/meta/library/hadoop/rhel7@latest.properties
2021-03-24 00:26:07,825 WARN [NM ContainerManager dispatcher]
runc.ImageTagToManifestV2Plugin
(ImageTagToManifestV2Plugin.java:getHdfsImageToHashReader(148)) - Did not load
the hdfs image to hash properties file, file doesn't exist
2021-03-24 00:26:07,828 WARN [NM ContainerManager dispatcher]
container.ContainerImpl (ContainerImpl.java:transition(1261)) - Failed to parse
resource-request
java.io.FileNotFoundException: File does not exist:
/runc-root/manifest/ha/hadoop/rhel7
{noformat}
Then I got this error in the NM when it was trying to resolve the tag. It added
the default {{metaNamespaceDir}} (which is library) into the path when looking
for the properties file. But when the CLI tool ran, it didn't add the
{{metaNamespaceDir}}. I didn't have the config set in my configs at all, so the
NM was using the conf default.
I'm not sure if I did anything wrong here or not, but it seems inconsistent to
me. Let me know what you think
> RunC container repository v2
> ----------------------------
>
> Key: YARN-10493
> URL: https://issues.apache.org/jira/browse/YARN-10493
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, yarn
> Affects Versions: 3.3.0
> Reporter: Craig Condit
> Assignee: Matthew Sharp
> Priority: Major
> Labels: pull-request-available
> Attachments: runc-container-repository-v2-design.pdf
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The current runc container repository design has scalability and usability
> issues which will likely limit widespread adoption. We should address this
> with a new, V2 layout.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
