[ https://issues.apache.org/jira/browse/YARN-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216956#comment-16216956 ]
Sunil G commented on YARN-7224: ------------------------------- Thanks [~leftnoteasy] for the effort. Few comments # {{YarnConfiguration.NVIDIA_DOCKER_PLUGIN_ENDPOINT}}. This is more vendor specific and hence its better to be in a separate config file altogether ? Something like a json file {code} { devices: gpu,fpga "gpu" : { "make" : "nvidia", "endpoint" : "http://localhost:3048" } } {code} # IN {{GpuDockerCommandPlugin#init}}, else is not needed as *if* condition is already giving a return statement. # To me {{GpuDockerCommandPlugin}} is mostly tightly coupled with nvidia. So could be rename it as NvidiaGpuDockerCommandPlugin # I could see a validation to ensure that volume always has ":ro". Do we need to validate this ? {{-volume=nvidia_driver_352.68:/usr/local/nvidia:ro}} # In {{getGpuIndexFromDeviceName}}, we night need to handle exception for parseInt. # Since we use Serializable for storing resource mapping, we do conversions back and to int and string. I think its better we define all gpu device number in string itself and access it via map. # In {{updateDockerRunCommand}}, why are we setting source and destination device as same ? {{dockerRunCommand.addDevice(value, value);}} # Could we return {{getAssignedGpus}} as Set<String>. Then in {{getGpuIndexFromDeviceName}}, we can give the device name and look for it. # one doubt {code} 248 // Cannot get all assigned Gpu devices from docker plugin output 249 if (foundGpuDevices < assignedResources.size()) { 250 // TODO: We can do better for this, instead directly compare device 251 // name, we should compare device's minor number with specified GPU 252 // minor number. 253 throw new ContainerExecutionException( 254 "Cannot get all assigned Gpu devices from docker plugin output"); {code} Instead of having {{foundGpuDevices}}, this should be a part of NM's resource capability vector. Ideally its better to associate device availabilty and its usage in NMs resource usage vector itself rather than recomputing here ? # > Support GPU isolation for docker container > ------------------------------------------ > > Key: YARN-7224 > URL: https://issues.apache.org/jira/browse/YARN-7224 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Wangda Tan > Assignee: Wangda Tan > Attachments: YARN-7224.001.patch, YARN-7224.002-wip.patch, > YARN-7224.003.patch, YARN-7224.004.patch, YARN-7224.005.patch > > > This patch is to address issues when docker container is being used: > 1. GPU driver and nvidia libraries: If GPU drivers and NV libraries are > pre-packaged inside docker image, it could conflict to driver and > nvidia-libraries installed on Host OS. An alternative solution is to detect > Host OS's installed drivers and devices, mount it when launch docker > container. Please refer to \[1\] for more details. > 2. Image detection: > From \[2\], the challenge is: > bq. Mounting user-level driver libraries and device files clobbers the > environment of the container, it should be done only when the container is > running a GPU application. The challenge here is to determine if a given > image will be using the GPU or not. We should also prevent launching > containers based on a Docker image that is incompatible with the host NVIDIA > driver version, you can find more details on this wiki page. > 3. GPU isolation. > *Proposed solution*: > a. Use nvidia-docker-plugin \[3\] to address issue #1, this is the same > solution used by K8S \[4\]. issue #2 could be addressed in a separate JIRA. > We won't ship nvidia-docker-plugin with out releases and we require cluster > admin to preinstall nvidia-docker-plugin to use GPU+docker support on YARN. > "nvidia-docker" is a wrapper of docker binary which can address #3 as well, > however "nvidia-docker" doesn't provide same semantics of docker, and it > needs to setup additional environments such as PATH/LD_LIBRARY_PATH to use > it. To avoid introducing additional issues, we plan to use > nvidia-docker-plugin + docker binary approach. > b. To address GPU driver and nvidia libraries, we uses nvidia-docker-plugin > \[3\] to create a volume which includes GPU-related libraries and mount it > when docker container being launched. Changes include: > - Instead of using {{volume-driver}}, this patch added {{docker volume > create}} command to c-e and NM Java side. The reason is {{volume-driver}} can > only use single volume driver for each launched docker container. > - Updated {{c-e}} and Java side, if a mounted volume is a named volume in > docker, skip checking file existence. (Named-volume still need to be added to > permitted list of container-executor.cfg). > c. To address isolation issue: > We found that, cgroup + docker doesn't work under newer docker version which > uses {{runc}} as default runtime. Setting {{--cgroup-parent}} to a cgroup > which include any {{devices.deny}} causes docker container cannot be launched. > Instead this patch passes allowed GPU devices via {{--device}} to docker > launch command. > References: > \[1\] https://github.com/NVIDIA/nvidia-docker/wiki/NVIDIA-driver > \[2\] https://github.com/NVIDIA/nvidia-docker/wiki/Image-inspection > \[3\] https://github.com/NVIDIA/nvidia-docker/wiki/nvidia-docker-plugin > \[4\] https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/ -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org