Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.
The "dineshs/IsolatingYarnAppsInDockerContainers" page has been changed by dineshs: https://wiki.apache.org/hadoop/dineshs/IsolatingYarnAppsInDockerContainers?action=diff&rev1=2&rev2=3 == Work items == - Realizing these benefits requires changes to both Docker and YARN. Several of the necessary Docker features for the above such as excluding intermediate data directory from copy-on-write file system and adding data node Unix socket from host into the container for short-circuit IO are already available. The following new pieces of work needs to be done. + Realizing these benefits requires changes to both Docker and YARN. Summary of those changes (existing and proposed) are listed below. - * '''YARN Docker executor''' - * An [[https://issues.apache.org/jira/browse/YARN-1964|initial patch]] of Docker executor. + * '''YARN''' + 1. [[https://issues.apache.org/jira/browse/YARN-1964|YARN-1964]] Initial support for running YARN applications within Docker containers through Docker Container Executor (DCE) + 2. [[https://issues.apache.org/jira/browse/YARN-2477|YARN-2477]] DCE must support secure mode + 3. [[https://issues.apache.org/jira/browse/YARN-2478|YARN-2478]] Nested containers should be supported + 4. [[https://issues.apache.org/jira/browse/YARN-2479|YARN-2479]] DCE must support handling of distributed cache + 5. [[https://issues.apache.org/jira/browse/YARN-2480|YARN-2480]] DCE must support user namespaces + 6. [[https://issues.apache.org/jira/browse/YARN-2482|YARN-2482]] DCE configuration - * Some of the Docker features below may only be made available via its REST endpoint. Docker executor should connect to it rather than shell out to invoke those functions. + 7. Some of the Docker features below may only be made available via its REST endpoint. DCE should connect to it rather than shell out to invoke those functions. - * '''Docker support for user namespaces''' to [[https://github.com/dotcloud/docker/pull/4572|map root user in the container]] to an unprivileged user on the host. Currently root in a Docker container has root privileges on the host. - * '''Container network configuration''' that allows the task and application master containers to talk to each other. The NAT'ed non-routable IP addresses assigned by Docker don't allow the task to reach the application master running in a container on a different host. Possible approaches to addressing this and relevant tickets are outlined [[dineshs/DockerNetworkingForYarnApps|here]]. - * '''Dynamic tuning of resource limits''' for [[https://github.com/dotcloud/docker/issues/6323|granular control over resources allocation]]. Docker currently does not allow changing container resources once created. + * '''Docker''' + 1. Support for user namespace to [[https://github.com/dotcloud/docker/pull/4572|map root user in the container]] to an unprivileged user on the host. Currently root in a Docker container has root privileges on the host. + 2. Container network configuration that allows the task and application master containers to talk to each other. The NAT'ed non-routable IP addresses assigned by Docker don't allow the task to reach the application master running in a container on a different host. Possible approaches to addressing this and relevant tickets are outlined [[dineshs/DockerNetworkingForYarnApps|here]]. + 3. Dynamic tuning of resource limits for [[https://github.com/dotcloud/docker/issues/6323|granular control over resources allocation]]. Docker currently does not allow changing container resources once created. +
