Repository: incubator-singa Updated Branches: refs/heads/master 5d076e526 -> 7c50e0313
SINGA-89 Add Docker support Docker is a Linux container which is fast to deploy and incurs small overhead. A Docker container functions like a light-weight virtual machine and runs in its own isolated environment. There are at least two benefits in building adding Docker support for SINGA: + Out of the box deployment of SINGA: the image, once built, contains the complete environment necessary to start SINGA. + Light-weight development and testing environment for distributed features. The image can be used to set up a test-bed consisting of many independent nodes (each node has its own IP address), without the need for cluster hardware. We add to Github a Dockerfile specifying how to build SINGA image, from which the user can construct the image by executing: sudo docker build -t singa/base . The user can choose another name beside singa/base . The build process can take a long time, but it needs to be done only at one host and the result image can then be copied to other hosts. We also add another Dockerfile which adds Mesos and Hadoop ontop of SINGA. This is closely related to SINGA-11. The image created from this Dockerfile is used to set up the distributed test bed. See the README.md for more details of the images and how to use them. IMPORTANT We assume that every host has Docker running. Two nodes on the epiC cluster (`ciidaa-c18` and `ciidaa-c19`) are set up with Docker and the pre-built images. IMPORTANT For now, the Dockerfile for building Mesos+Hadoop+SINGA will pull the latest from ug93tad's (Anh's) SINGA-11 branch. This pull step is removed after SINGA-11 is merged to the project's master branch. Project: http://git-wip-us.apache.org/repos/asf/incubator-singa/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-singa/commit/1f513ec1 Tree: http://git-wip-us.apache.org/repos/asf/incubator-singa/tree/1f513ec1 Diff: http://git-wip-us.apache.org/repos/asf/incubator-singa/diff/1f513ec1 Branch: refs/heads/master Commit: 1f513ec1e6ea75c8b1dcd0582022a7669bab595e Parents: 497dafc Author: Anh Dinh <[email protected]> Authored: Thu Jun 25 08:30:46 2015 -0700 Committer: Anh Dinh <[email protected]> Committed: Sun Oct 18 12:03:13 2015 +0800 ---------------------------------------------------------------------- tool/docker/README.md | 181 +++++++++++++++++++++++++++++++++ tool/docker/mesos/.bashrc | 10 ++ tool/docker/mesos/.vimrc | 7 ++ tool/docker/mesos/Dockerfile | 19 ++++ tool/docker/mesos/core-site.xml | 24 +++++ tool/docker/mesos/hdfs-site.xml | 34 +++++++ tool/docker/mesos/install.sh | 16 +++ tool/docker/mesos/mapred-site.xml | 22 ++++ tool/docker/mesos/yarn-site.xml | 32 ++++++ tool/docker/singa/.bashrc | 8 ++ tool/docker/singa/.vimrc | 7 ++ tool/docker/singa/Dockerfile | 15 +++ tool/docker/singa/ssh.conf | 5 + 13 files changed, 380 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/1f513ec1/tool/docker/README.md ---------------------------------------------------------------------- diff --git a/tool/docker/README.md b/tool/docker/README.md new file mode 100644 index 0000000..cfc3192 --- /dev/null +++ b/tool/docker/README.md @@ -0,0 +1,181 @@ +This guide explains how to set up a development environment for SINGA. A cluster environment can be launched over one or multiple physical nodes. + +1. [Build SINGA base](#build_base) +2. [Build SINGA with Mesos and Hadoop](#build_mesos) +3. [Pre-built images](#pre_built) +4. [Launch stand-alone SINGA](#launch_stand_alone) +5. [Launch pseudo-distributed SINGA on one node](#launch_pseudo) +6. [Launch fully distributed SINGA on multiple nodes](#launch_distributed) + +--- + +<a name="build_base"></a> +#### Build SINGA base image + +```` +$ cd tool/docker/singa +$ sudo docker built -t singa/base . +$ sudo docker images +REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE +singa/base latest XXXX XXX 2.01 GB +```` + +The result is the image containing a built version of SINGA. + +  + + *Figure 1. singa/base Docker image, containing library dependencies and SINGA built from source.* + +-- + +<a name="build_mesos"></a> +#### Build SINGA with Mesos and Hadoop +```` +$ cd tool/docker/mesos +$ sudo docker build -t singa/mesos . +$ sudo docker images +REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE +singa/mesos latest XXXX XXX 4.935 GB +```` +  + + *Figure 2. singa/mesos Docker image, containing Hadoop and Mesos built on +top of SINGA. The default namenode address for Hadoop is `node0:9000`* + +-- + +<a name="pre_built"></a> +#### Pre-built images on epiC cluster +For users with access to the `epiC` cluster, there are pre-built and loaded Docker images at the following nodes: + + ciidaa-c18 + ciidaa-c19 + +The available images at those nodes are: +```` +REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE +singa/base latest XXXX XXX 2.01 GB +singa/mesos latest XXXX XXX 4.935 GB +weaveworks/weaveexec 1.1.1 XXXX 11 days ago 57.8 MB +weaveworks/weave 1.1.1 XXXX 11 days ago 17.56 MB +```` + +-- + +<a name="launch_stand_alone"></a> +#### Launch stand-alone SINGA +To launch a test environment for a single-node SINGA training, simply start a +container from `singa/base` image. The following starts a container called +`XYZ`, then launches a shell in the container: + +```` +$ sudo docker run -dt --name XYZ singa/base /usr/bin/supervisord +$ sudo docker exec -it XYZ /bin/bash +```` + +  + + *Figure 3. Launch SINGA in stand-alone mode: single node training* + +Inside the launched container, the SINGA source directory can be found at `/root/incubator-singa` + +-- + +<a name="launch_pseudo"></a> +#### Launch SINGA on pseudo-distributed mode (single node) +To simulate a distributed environment on a single node, one can repeat the +previous step multiple times, each time giving a different name to the +container. Network connections between these containers are already supported, +thus SINGA instances/nodes in these container can readily communicate with each +other. + +The previous approach requires the user to start SINGA instances individually +at each container. Although there's a bash script for that, we provide a better +way. In particular, multiple containers can be started from `singa/mesos` image +which already bundles Mesos and Hadoop with SINGA. Using Mesos makes it easy to +launch, stop and monitor the distributed execution from a single container. +Figure 4 shows `N+1` containers running concurrently at the local host. + +```` +# sudo docker run -dt --name node0 singa/mesos /usr/bin/supervisord +# sudo docker run -dt --name node1 singa/mesos /usr/bin/supervisord +... +```` +  + + *Figure 4. Launch SINGA in pseudo-distributed mode : multiple SINGA nodes over one single machine* + +**Starting SINGA distributed training** + +Refer to the [Mesos +guide](https://github.com/ug93tad/incubator-singa/blob/SINGA-11/tool/mesos/README.md) +for details of how to start training with multiple SINGA instances. + +**Important:** the container that assumes the role of Hadoop's namenode (and often Mesos's and Zookeeper's mater node as well) **must** be named `node0`. Otherwise, the user must log in to individual containers and change the Hadoop configuration separately. + +-- + +<a name="launch_distributed"></a> +#### Launch SINGA on fully distributed mode (multiple nodes) +The previous section has explained how to start a distributed environment on a +single node. But running many containers on one node does not scale. When there +are multiple physical hosts available, it is better to distribute the +containers over them. + +The only extra requirement for the fully distributed mode, as compared with the +pseudo distributed mode, is that the containers from different hosts are able +to transparently communicate with each other. In the pseudo distributed mode, +the local docker engine takes care of such communication. Here, we rely on +[Weave](http://weave.works/guides/weave-docker-ubuntu-simple.html) to make the +communication transparent. The resulting architecture is shown below. + +  + + *Figure 5. Launch SINGA in fully distributed mode: multiple SINGA nodes over multiple machines* + +**Install Weave at all hosts** +``` +$ curl -L git.io/weave -o /usr/local/bin/weave +$ chmod a+x /usr/local/bin/weave +``` + +**Starting Weave** + +Suppose `node0` will be launched at host with IP `111.222.111.222`. + +* At host `111.222.111.222`: +```` +$ weave launch +$ eval "$(weave env)" //if there's error, do `sudo -s` and try again +```` +* At other hosts: +```` +$ weave launch 111.222.111.222 +$ eval "$(weave env)" //if there's error, do `sudo -s` and try again +```` + +**Starting containers** + +The user logs in to each host and starts the container (same as in [pseudo-distributed](#launch_pseudo) mode). Note that container acting as the head node of the cluster must be named `node0` (and be running at the host with IP `111.222.111.222`, for example). + +**_Important_:** when there are other containers sharing the same host as `node0`, say `node1` and `node2` for example, +there're additional changes to be made to `node1` and `node2`. Particularly, log in to each container and edit +`/etc/hosts` file: + +```` +# modified by weave +... +X.Y.Z node0 node0.bridge //<- REMOVE this line +.. +```` +This is to ensure that name resolutions (of `node0`'s address) from `node1` and `node2` are correct. By default, +containers of the same host resolves each other's addresses via the Docker bridge. Instead, we want they to use +addressed given by Weave. + + +**Starting SINGA distributed training** + +Refer to the [Mesos +guide](https://github.com/ug93tad/incubator-singa/blob/SINGA-11/tool/mesos/README.md) +for details of how to start training with multiple SINGA instances. + http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/1f513ec1/tool/docker/mesos/.bashrc ---------------------------------------------------------------------- diff --git a/tool/docker/mesos/.bashrc b/tool/docker/mesos/.bashrc new file mode 100644 index 0000000..4a649ac --- /dev/null +++ b/tool/docker/mesos/.bashrc @@ -0,0 +1,10 @@ +export LIBRARY_PATH=/opt/OpenBLAS/lib:$LIBRARY_PATH +export LD_LIBRARY_PATH=/opt/OpenBLAS/lib:/usr/local/lib:$LD_LIBRARY_PATH +export CPLUS_INCLUDE_PATH=/opt/OpenBLAS/include:$CPLUS_INCLUDE_PATH +export PATH=$PATH:/opt/jdk1.8.0_60/bin:/opt/bin +export HADOOP_HOME=/opt/hadoop-2.6.0 +export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin +export JAVA_HOME=/opt/jdk1.8.0_60 +alias ls="ls --color=always" +export SINGA_HOME=/root/incubator-singa +export PATH=$PATH:$SINGA_HOME/bin http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/1f513ec1/tool/docker/mesos/.vimrc ---------------------------------------------------------------------- diff --git a/tool/docker/mesos/.vimrc b/tool/docker/mesos/.vimrc new file mode 100644 index 0000000..2269792 --- /dev/null +++ b/tool/docker/mesos/.vimrc @@ -0,0 +1,7 @@ +set autoindent " always set autoindenting on +set smartindent " use smart indent if there is no indent file +set tabstop=2 " <tab> inserts 4 spaces +set shiftwidth=2 " but an indent level is 2 spaces wide. +set softtabstop=2 " <BS> over an autoindent deletes both spaces. +"set expandtab " Use spaces, not tabs, for autoindent/tab key. +set shiftround " rounds indent to a multiple of shiftwidth http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/1f513ec1/tool/docker/mesos/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/docker/mesos/Dockerfile b/tool/docker/mesos/Dockerfile new file mode 100644 index 0000000..2d4871c --- /dev/null +++ b/tool/docker/mesos/Dockerfile @@ -0,0 +1,19 @@ +#build Mesos+Hadoop on top of SINGA +FROM singa/base +MAINTAINER Anh Dinh<[email protected]> + +COPY .bashrc /root/.bashrc +COPY install.sh /opt/install.sh + +RUN cd /opt && source ./install.sh + +RUN apt-get -y install curl cmake libxml2 libxml2-dev uuid-dev protobuf-compiler libprotobuf-dev libgsasl7-dev libkrb5-dev libboost1.54-all-dev + +RUN cd /opt && source /root/.bashrc && wget -c http://www.eu.apache.org/dist/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz && tar -zxvf hadoop-2.6.0.tar.gz && git clone https://github.com/PivotalRD/libhdfs3.git && cd libhdfs3 && mkdir build && cd build && cmake ../ && make && make install + +#to be removed after SINGA-11 is merged (pulling from Anh's branch instead of pulling from the master) +RUN source ~/.bashrc && mkdir /root/mesos && cd /root/mesos && git clone https://github.com/ug93tad/incubator-singa && cd incubator-singa && git checkout SINGA-11 && cp -r tool/mesos /root/incubator-singa/tool/ && cd /root/incubator-singa/tool/mesos && make + +COPY *.xml /opt/hadoop-2.6.0/etc/hadoop/ + +RUN ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa && cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys && echo 'StrictHostKeyChecking no' >> ~/.ssh/config http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/1f513ec1/tool/docker/mesos/core-site.xml ---------------------------------------------------------------------- diff --git a/tool/docker/mesos/core-site.xml b/tool/docker/mesos/core-site.xml new file mode 100644 index 0000000..49a39ca --- /dev/null +++ b/tool/docker/mesos/core-site.xml @@ -0,0 +1,24 @@ +<?xml version="1.0" encoding="UTF-8"?> +<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +<!-- Put site-specific property overrides in this file. --> + +<configuration> + <property> + <name>fs.default.name</name> + <value>hdfs://node0:9000</value> + </property> +</configuration> http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/1f513ec1/tool/docker/mesos/hdfs-site.xml ---------------------------------------------------------------------- diff --git a/tool/docker/mesos/hdfs-site.xml b/tool/docker/mesos/hdfs-site.xml new file mode 100644 index 0000000..2e0f4c6 --- /dev/null +++ b/tool/docker/mesos/hdfs-site.xml @@ -0,0 +1,34 @@ +<?xml version="1.0" encoding="UTF-8"?> +<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +<!-- Put site-specific property overrides in this file. --> + +<configuration> +<property> + <name>dfs.replication</name> + <value>1</value> + </property> + +<property> + <name>dfs.name.dir</name> + <value>file:///var/lib/hadoop/hdfs/namenode</value> + </property> + + <property> + <name>dfs.data.dir</name> + <value>file:///var/lib/hadoop/hdfs/datanode</value> + </property> +</configuration> http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/1f513ec1/tool/docker/mesos/install.sh ---------------------------------------------------------------------- diff --git a/tool/docker/mesos/install.sh b/tool/docker/mesos/install.sh new file mode 100644 index 0000000..560fbab --- /dev/null +++ b/tool/docker/mesos/install.sh @@ -0,0 +1,16 @@ +#!/bin/bash +source ~/.bashrc +#download +cd /opt +wget -c http://archive.apache.org/dist/mesos/0.22.0/mesos-0.22.0.tar.gz +wget https://www.comp.nus.edu.sg/~dinhtta/files/mesos_patch +tar -zxvf mesos-0.22.0.tar.gz + +#patch and install mesos +cd /opt/mesos-0.22.0 +patch -p5 < ../mesos_patch +mkdir build; cd build +../configure +make +sudo make install + http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/1f513ec1/tool/docker/mesos/mapred-site.xml ---------------------------------------------------------------------- diff --git a/tool/docker/mesos/mapred-site.xml b/tool/docker/mesos/mapred-site.xml new file mode 100644 index 0000000..42bb81c --- /dev/null +++ b/tool/docker/mesos/mapred-site.xml @@ -0,0 +1,22 @@ +<?xml version="1.0"?> +<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +<!-- Put site-specific property overrides in this file. --> + +<configuration> + <property> + <name>mapreduce.framework.name</name> + <value>yarn</value> + </property> +</configuration> http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/1f513ec1/tool/docker/mesos/yarn-site.xml ---------------------------------------------------------------------- diff --git a/tool/docker/mesos/yarn-site.xml b/tool/docker/mesos/yarn-site.xml new file mode 100644 index 0000000..c137af9 --- /dev/null +++ b/tool/docker/mesos/yarn-site.xml @@ -0,0 +1,32 @@ +<?xml version="1.0"?> +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> +<configuration> + +<!-- Site specific YARN configuration properties --> + <property> + <name>yarn.nodemanager.aux-services</name> + <value>mapreduce_shuffle</value> + </property> + + <property> + <name>yarn.resourcemanager.hostname</name> + <value>node0</value> + </property> + + <property> + <name>yarn.resourcemanager.bind-host</name> + <value>0.0.0.0</value> + </property> +</configuration> http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/1f513ec1/tool/docker/singa/.bashrc ---------------------------------------------------------------------- diff --git a/tool/docker/singa/.bashrc b/tool/docker/singa/.bashrc new file mode 100644 index 0000000..40ebe41 --- /dev/null +++ b/tool/docker/singa/.bashrc @@ -0,0 +1,8 @@ +export LIBRARY_PATH=/opt/OpenBLAS/lib:$LIBRARY_PATH +export LD_LIBRARY_PATH=/opt/OpenBLAS/lib:/usr/local/lib:$LD_LIBRARY_PATH +export CPLUS_INCLUDE_PATH=/opt/OpenBLAS/include:$CPLUS_INCLUDE_PATH +export PATH=$PATH:/opt/jdk1.8.0_60/bin:/opt/bin +export JAVA_HOME=/opt/jdk1.8.0_60 +export SINGA_HOME=/root/incubator-singa +export PATH=$PATH:$SINGA_HOME/bin +alias ls="ls --color=always" http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/1f513ec1/tool/docker/singa/.vimrc ---------------------------------------------------------------------- diff --git a/tool/docker/singa/.vimrc b/tool/docker/singa/.vimrc new file mode 100644 index 0000000..2269792 --- /dev/null +++ b/tool/docker/singa/.vimrc @@ -0,0 +1,7 @@ +set autoindent " always set autoindenting on +set smartindent " use smart indent if there is no indent file +set tabstop=2 " <tab> inserts 4 spaces +set shiftwidth=2 " but an indent level is 2 spaces wide. +set softtabstop=2 " <BS> over an autoindent deletes both spaces. +"set expandtab " Use spaces, not tabs, for autoindent/tab key. +set shiftround " rounds indent to a multiple of shiftwidth http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/1f513ec1/tool/docker/singa/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/docker/singa/Dockerfile b/tool/docker/singa/Dockerfile new file mode 100644 index 0000000..b62162f --- /dev/null +++ b/tool/docker/singa/Dockerfile @@ -0,0 +1,15 @@ +#build singa from scratch +FROM ubuntu:14.04 +MAINTAINER Anh Dinh<[email protected]> +RUN rm /bin/sh && ln -s /bin/bash /bin/sh + +RUN apt-get update && apt-get -y install g++-4.8 build-essential git vim wget zip automake libtool man python-dev python-boto libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev openssh-server supervisor + +RUN cd /opt && wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u60-b27/jdk-8u60-linux-x64.tar.gz && cd /opt && tar -zxvf jdk-8u60-linux-x64.tar.gz && rm -rf jdk-8u60-linux-x64.tar.gz + +COPY ssh.conf /etc/supervisor/conf.d/ +COPY .bashrc /root/.bashrc +copy .vimrc /root/.vimrc + +RUN cd && source /root/.bashrc && git clone https://github.com/apache/incubator-singa.git && cd incubator-singa && ./thirdparty/install.sh all && cd thirdparty/ && wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz && tar -zxvf protobuf-2.5.0.tar.gz && cd protobuf-2.5.0 && ./configure && make && make install && cd ../../ && cp thirdparty/install.sh . && rm -rf thirdparty/* && mv install.sh thirdparty/ && ./thirdparty/install.sh zookeeper && ./autogen.sh && ./configure && make && make install && cd examples/cifar10 && mv Makefile.example Makefile && make download && make create && service ssh start + http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/1f513ec1/tool/docker/singa/ssh.conf ---------------------------------------------------------------------- diff --git a/tool/docker/singa/ssh.conf b/tool/docker/singa/ssh.conf new file mode 100644 index 0000000..ac04d7e --- /dev/null +++ b/tool/docker/singa/ssh.conf @@ -0,0 +1,5 @@ +[supervisord] +nodaemon=true + +[program:ssh] +command=/usr/sbin/sshd -D
