[
https://issues.apache.org/jira/browse/YARN-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16191803#comment-16191803
]
Wangda Tan commented on YARN-7224:
----------------------------------
YARN-6620 added support of GPU isolation in NM side, which only supports
non-docker containers. We need to add support to help docker containers
launched by YARN can utilize GPUs.
Currently, YARN launches docker containers inside cgroups. After
YARN-6852/YARN-6620, we have finished cgroup-based GPU isolation logics. This
patch is to address issues when docker container is being used.
There're several issues:
1. GPU driver and nvidia libraries: If GPU drivers and NV libraries are
pre-packaged inside docker image, it could conflict to driver and
nvidia-libraries installed on Host OS. An alternative solution is to detect
Host OS's installed drivers and devices, mount it when launch docker container.
Please refer to \[1\] for more details.
2. Image detection:
>From \[2\], the challenge is:
bq. Mounting user-level driver libraries and device files clobbers the
environment of the container, it should be done only when the container is
running a GPU application. The challenge here is to determine if a given image
will be using the GPU or not. We should also prevent launching containers based
on a Docker image that is incompatible with the host NVIDIA driver version, you
can find more details on this wiki page.
3. GPU isolation:
We have already done this in YARN-6852/YARN-6620.
*Proposal:*
My plan is to use nvidia-docker-plugin \[3\] to address #1, this is the same
solution used by K8S \[4\]. #2 could be addressed in a separate JIRA.
We won't ship nvidia-docker-plugin with out releases and we require cluster
admin to preinstall nvidia-docker-plugin to use GPU+docker support on YARN.
"nvidia-docker" is a wrapper of docker binary which can address #3 as well,
however "nvidia-docker" doesn't provide same semantics of docker, and it needs
to setup additional environments such as PATH/LD_LIBRARY_PATH to use it. To
avoid introducing additional issues, we plan to use nvidia-docker-plugin +
docker binary approach.
\[1\] https://github.com/NVIDIA/nvidia-docker/wiki/NVIDIA-driver
\[2\] https://github.com/NVIDIA/nvidia-docker/wiki/Image-inspection
\[3\] https://github.com/NVIDIA/nvidia-docker/wiki/nvidia-docker-plugin
\[4\] https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/
> Support GPU isolation for docker container
> ------------------------------------------
>
> Key: YARN-7224
> URL: https://issues.apache.org/jira/browse/YARN-7224
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Wangda Tan
> Assignee: Wangda Tan
>
> YARN-6620 added support of GPU isolation in NM side, which only supports
> non-docker containers. We need to add support to help docker containers
> launched by YARN can utilize GPUs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
