SINGA-276 Create docker images Created the dockerfiles for tag 'devel' and 'devel-cuda'. They are described in doc/en/docs/docker.md file. Instructions and naming rules of docker image tags are also included in that file.
Project: http://git-wip-us.apache.org/repos/asf/incubator-singa/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-singa/commit/c3ce75e9 Tree: http://git-wip-us.apache.org/repos/asf/incubator-singa/tree/c3ce75e9 Diff: http://git-wip-us.apache.org/repos/asf/incubator-singa/diff/c3ce75e9 Branch: refs/heads/master Commit: c3ce75e99346ff2b4b403d344080b5f261cc29a8 Parents: 8dac86c Author: wangwei <[email protected]> Authored: Fri Dec 23 10:06:24 2016 +0800 Committer: wangwei <[email protected]> Committed: Fri Jan 13 13:35:16 2017 +0800 ---------------------------------------------------------------------- doc/en/docs/docker.md | 58 ++++++++ tool/docker/README.md | 189 --------------------------- tool/docker/devel/Dockerfile | 20 +++ tool/docker/devel/cuda/Dockerfile | 23 ++++ tool/docker/mesos/.bashrc | 32 ----- tool/docker/mesos/Dockerfile | 40 ------ tool/docker/mesos/core-site.xml | 24 ---- tool/docker/mesos/hdfs-site.xml | 34 ----- tool/docker/mesos/install.sh | 35 ----- tool/docker/mesos/mapred-site.xml | 22 ---- tool/docker/mesos/yarn-site.xml | 32 ----- tool/docker/singa/.bashrc | 30 ----- tool/docker/singa/Dockerfile | 34 ----- tool/docker/singa/Dockerfile_gpu | 80 ------------ tool/docker/singa/ssh.conf | 5 - tool/mesos/docker/README.md | 189 +++++++++++++++++++++++++++ tool/mesos/docker/mesos/.bashrc | 32 +++++ tool/mesos/docker/mesos/Dockerfile | 40 ++++++ tool/mesos/docker/mesos/core-site.xml | 24 ++++ tool/mesos/docker/mesos/hdfs-site.xml | 34 +++++ tool/mesos/docker/mesos/install.sh | 35 +++++ tool/mesos/docker/mesos/mapred-site.xml | 22 ++++ tool/mesos/docker/mesos/yarn-site.xml | 32 +++++ tool/mesos/docker/singa/.bashrc | 30 +++++ tool/mesos/docker/singa/Dockerfile | 34 +++++ tool/mesos/docker/singa/Dockerfile_gpu | 80 ++++++++++++ tool/mesos/docker/singa/ssh.conf | 5 + 27 files changed, 658 insertions(+), 557 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/doc/en/docs/docker.md ---------------------------------------------------------------------- diff --git a/doc/en/docs/docker.md b/doc/en/docs/docker.md new file mode 100644 index 0000000..8e5743e --- /dev/null +++ b/doc/en/docs/docker.md @@ -0,0 +1,58 @@ +# Docker Images + + +## Available tags + +* `devel`, with SINGA and the development packages installed on Ubuntu16.04 (no GPU) +* `devel-cuda`, with SINGA, CUDA8.0, CUDNN5, and other development packages installed on Ubuntu16.04 + +## Use the existing Docker images + +Users can pull the Docker images from Dockerhub via + + docker pull apache/singa:devel + # or + docker pull apache/singa:devel-cuda + +Run the docker container using + + docker run -it apache/singa:devel /bin/bash + # or + docker run -it apache/singa:devel-cuda /bin/bash + +The latest SINGA code is under the `incubator-singa` folder. + +## Create new Docker images from Dockerfile + +New Docker images could be created by executing the following command within the +Dockerfile folder, e.g., tool/docker/devel/ + + docker build -t singa:<TAG> -f Dockerfile + +The `<TAG>` is named as + + devel|runtime[-OS][-CUDA|OPENCL][-CUDNN] + +* devel: development images with all dependent libs' header files installed and SINGA's source code; runtime: the minimal images which can run SINGA programs. +* OS: ubuntu, ubuntu14.04, centos, centos6 +* CUDA: cuda, cuda8.0, cuda7.0 +* CUDNN: cudnn, cudnn5, cudnn4 +* OPENCL: opencl, opencl1.2 + +By default, if the version is not included in the tag, the latest stable version is used. +The default OS is Ubuntu. The version is the latest stable version (e.g., 16.04 for now). +For -cuda version, the **cudnn** is included by default. Their versions are also the latest stable version, i.e., cuda8.0 and cudnn5 for now. + +Here are some example tags, + +`devel`, `devel-cuda`, `runtime`, `runtime-cuda`, `devel-centos7-cuda`, `devel-ubuntu14.04`, `devel-ubuntu14.04-cuda7.5-cudnn4` + +Please follow the existing Dockefiles under tool/docker/ to create other Dockefiles. +The folder structure is like + + level1: devel|runtime + level2: Dockerfile, OS + level3: Dockerfile, CUDA|OPENCL + level4: CUDNN + +For example, the path of the Dockerfile for `devel-cuda` is `tool/docker/devel/cuda/Dockerfile`. http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/README.md ---------------------------------------------------------------------- diff --git a/tool/docker/README.md b/tool/docker/README.md deleted file mode 100644 index fd33e7e..0000000 --- a/tool/docker/README.md +++ /dev/null @@ -1,189 +0,0 @@ -This guide explains how to set up a development environment for SINGA. A cluster environment can be launched over one or multiple physical nodes. - -1. [Build SINGA base](#build_base) -2. [Build SINGA with Mesos and Hadoop](#build_mesos) -3. [Pre-built images](#pre_built) -4. [Launch and stop SINGA (stand alone mode)](#launch_stand_alone) -5. [Launch pseudo-distributed SINGA on one node](#launch_pseudo) -6. [Launch fully distributed SINGA on multiple nodes](#launch_distributed) - ---- - -<a name="build_base"></a> -#### Build SINGA base image - -```` -$ cd tool/docker/singa -$ sudo docker build -t singa/base . -$ sudo docker images -REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE -singa/base latest XXXX XXX 2.01 GB -```` - -The result is the image containing a built version of SINGA. - -  - - *Figure 1. singa/base Docker image, containing library dependencies and SINGA built from source.* - --- - -<a name="build_mesos"></a> -#### Build SINGA with Mesos and Hadoop -```` -$ cd tool/docker/mesos -$ sudo docker build -t singa/mesos . -$ sudo docker images -REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE -singa/mesos latest XXXX XXX 4.935 GB -```` -  - - *Figure 2. singa/mesos Docker image, containing Hadoop and Mesos built on -top of SINGA. The default namenode address for Hadoop is `node0:9000`* - -**Notes** A common failure observed during the build process is caused by network failure occuring when downloading dependencies. Simply re-run the build command. - --- - -<a name="pre_built"></a> -#### Pre-built images on epiC cluster -For users with access to the `epiC` cluster, there are pre-built and loaded Docker images at the following nodes: - - ciidaa-c18 - ciidaa-c19 - -The available images at those nodes are: -```` -REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE -singa/base latest XXXX XXX 2.01 GB -singa/mesos latest XXXX XXX 4.935 GB -weaveworks/weaveexec 1.1.1 XXXX 11 days ago 57.8 MB -weaveworks/weave 1.1.1 XXXX 11 days ago 17.56 MB -```` - --- - -<a name="launch_stand_alone"></a> -#### Launch and stop SINGA in stand-alone mode -To launch a test environment for a single-node SINGA training, simply start a container from `singa/base` image. The following starts a container called -`XYZ`, then launches a shell in the container: - -```` -$ sudo docker run -dt --name XYZ singa/base /usr/bin/supervisord -$ sudo docker exec -it XYZ /bin/bash -```` - -  - - *Figure 3. Launch SINGA in stand-alone mode: single node training* - -Inside the launched container, the SINGA source directory can be found at `/root/incubator-singa`. - -**Stopping the container** - -```` -$ sudo docker stop XYZ -$ sudo docker rm ZYZ -```` - --- - -<a name="launch_pseudo"></a> -#### Launch SINGA on pseudo-distributed mode (single node) -To simulate a distributed environment on a single node, one can repeat the -previous step multiple times, each time giving a different name to the -container. Network connections between these containers are already supported, -thus SINGA instances/nodes in these container can readily communicate with each -other. - -The previous approach requires the user to start SINGA instances individually -at each container. Although there's a bash script for that, we provide a better -way. In particular, multiple containers can be started from `singa/mesos` image -which already bundles Mesos and Hadoop with SINGA. Using Mesos makes it easy to -launch, stop and monitor the distributed execution from a single container. -Figure 4 shows `N+1` containers running concurrently at the local host. - -```` -# sudo docker run -dt --name node0 singa/mesos /usr/bin/supervisord -# sudo docker run -dt --name node1 singa/mesos /usr/bin/supervisord -... -```` -  - - *Figure 4. Launch SINGA in pseudo-distributed mode : multiple SINGA nodes over one single machine* - -**Starting SINGA distributed training** - -Refer to the [Mesos -guide](https://github.com/ug93tad/incubator-singa/blob/SINGA-11/tool/mesos/README.md) -for details of how to start training with multiple SINGA instances. - -**Important:** the container that assumes the role of Hadoop's namenode (and often Mesos's and Zookeeper's mater node as well) **must** be named `node0`. Otherwise, the user must log in to individual containers and change the Hadoop configuration separately. - --- - -<a name="launch_distributed"></a> -#### Launch SINGA on fully distributed mode (multiple nodes) -The previous section has explained how to start a distributed environment on a -single node. But running many containers on one node does not scale. When there -are multiple physical hosts available, it is better to distribute the -containers over them. - -The only extra requirement for the fully distributed mode, as compared with the -pseudo distributed mode, is that the containers from different hosts are able -to transparently communicate with each other. In the pseudo distributed mode, -the local docker engine takes care of such communication. Here, we rely on -[Weave](http://weave.works/guides/weave-docker-ubuntu-simple.html) to make the -communication transparent. The resulting architecture is shown below. - -  - - *Figure 5. Launch SINGA in fully distributed mode: multiple SINGA nodes over multiple machines* - -**Install Weave at all hosts** -``` -$ curl -L git.io/weave -o /usr/local/bin/weave -$ chmod a+x /usr/local/bin/weave -``` - -**Starting Weave** - -Suppose `node0` will be launched at host with IP `111.222.111.222`. - -* At host `111.222.111.222`: -```` -$ weave launch -$ eval "$(weave env)" //if there's error, do `sudo -s` and try again -```` -* At other hosts: -```` -$ weave launch 111.222.111.222 -$ eval "$(weave env)" //if there's error, do `sudo -s` and try again -```` - -**Starting containers** - -The user logs in to each host and starts the container (same as in [pseudo-distributed](#launch_pseudo) mode). Note that container acting as the head node of the cluster must be named `node0` (and be running at the host with IP `111.222.111.222`, for example). - -**_Important_:** when there are other containers sharing the same host as `node0`, say `node1` and `node2` for example, -there're additional changes to be made to `node1` and `node2`. Particularly, log in to each container and edit -`/etc/hosts` file: - -```` -# modified by weave -... -X.Y.Z node0 node0.bridge //<- REMOVE this line -.. -```` -This is to ensure that name resolutions (of `node0`'s address) from `node1` and `node2` are correct. By default, -containers of the same host resolves each other's addresses via the Docker bridge. Instead, we want they to use -addressed given by Weave. - - -**Starting SINGA distributed training** - -Refer to the [Mesos -guide](https://github.com/ug93tad/incubator-singa/blob/SINGA-11/tool/mesos/README.md) -for details of how to start training with multiple SINGA instances. - http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/devel/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/docker/devel/Dockerfile b/tool/docker/devel/Dockerfile new file mode 100644 index 0000000..c8827cc --- /dev/null +++ b/tool/docker/devel/Dockerfile @@ -0,0 +1,20 @@ +# Base unbuntu 16.04 image +FROM ubuntu:latest + +MAINTAINER incubator-singa [email protected] + +# install dependencies +RUN apt-get update \ + && apt-get install -y --no-install-recommends build-essential git cmake libprotobuf-dev libopenblas-dev protobuf-compiler python-dev python-pip swig wget\ + && apt-get clean && apt-get autoremove && apt-get autoclean \ + && rm -rf /var/lib/apt/lists/* \ + && pip --no-cache-dir install -U pip numpy setuptools + +# set environment +ENV CPLUS_INCLUDE_PATH /usr/local/lib/python2.7/dist-packages/numpy/core/include:${CPLUS_INCLUDE_PATH} + +# download singa source +RUN git clone https://github.com/apache/incubator-singa.git + +# compile singa and install pysinga +RUN cd incubator-singa && mkdir build && cd build && cmake .. && make && cd python && pip install . && cd ../../ && rm -rf build http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/devel/cuda/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/docker/devel/cuda/Dockerfile b/tool/docker/devel/cuda/Dockerfile new file mode 100644 index 0000000..76772ba --- /dev/null +++ b/tool/docker/devel/cuda/Dockerfile @@ -0,0 +1,23 @@ +# Base unbuntu 16.04 image from nvidia/cuda +# Change tags to build with different cuda/cudnn versions: +# FROM nvidia/cuda:8.0-cudnn5-devel-ubuntu16.04 +FROM nvidia/cuda:8.0-cudnn5-devel-ubuntu16.04 + +MAINTAINER incubator-singa [email protected] + +# install dependencies +RUN apt-get update \ + && apt-get install -y --no-install-recommends git g++ cmake libprotobuf-dev libopenblas-dev protobuf-compiler python-dev python-pip swig wget\ + && apt-get clean && apt-get autoremove && apt-get autoclean \ + && rm -rf /var/lib/apt/lists/* \ + && pip --no-cache-dir install -U pip numpy setuptools + +# set environment +ENV CPLUS_INCLUDE_PATH /usr/local/lib/python2.7/dist-packages/numpy/core/include:${CPLUS_INCLUDE_PATH} +ENV CMAKE_INCLUDE_PATH /usr/local/cuda/include:${CMAKE_INCLUDE_PATH} +ENV CMAKE_LIBRARY_PATH /usr/local/cuda/lib64:${CMAKE_LIBRARY_PATH} + +# download singa source +RUN git clone https://github.com/apache/incubator-singa.git + +RUN cd incubator-singa && mkdir build && cd build && cmake -DUSE_CUDA=ON .. && make && cd python && pip install . && cd ../../ && rm -rf build http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/mesos/.bashrc ---------------------------------------------------------------------- diff --git a/tool/docker/mesos/.bashrc b/tool/docker/mesos/.bashrc deleted file mode 100644 index c796a27..0000000 --- a/tool/docker/mesos/.bashrc +++ /dev/null @@ -1,32 +0,0 @@ -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ - -export LIBRARY_PATH=/opt/OpenBLAS/lib:$LIBRARY_PATH -export LD_LIBRARY_PATH=/opt/OpenBLAS/lib:/usr/local/lib:$LD_LIBRARY_PATH -export CPLUS_INCLUDE_PATH=/opt/OpenBLAS/include:$CPLUS_INCLUDE_PATH -export PATH=/opt/jdk1.8.0_60/bin:/opt/bin:$PATH -export HADOOP_HOME=/opt/hadoop-2.6.0 -export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin -export JAVA_HOME=/opt/jdk1.8.0_60 -alias ls="ls --color=always" -# some more ls aliases -alias ll="ls -alF" -alias la="ls -A" -alias l="ls -CF" -export SINGA_HOME=/root/incubator-singa -export PATH=$PATH:$SINGA_HOME/bin http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/mesos/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/docker/mesos/Dockerfile b/tool/docker/mesos/Dockerfile deleted file mode 100644 index 83142b3..0000000 --- a/tool/docker/mesos/Dockerfile +++ /dev/null @@ -1,40 +0,0 @@ -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ - -#build Mesos+Hadoop on top of SINGA -FROM singa/base -MAINTAINER Anh Dinh<[email protected]> - -COPY incubator-singa/tool/docker/mesos/.bashrc /root/.bashrc -COPY incubator-singa/tool/docker/mesos/install.sh /opt/install.sh -COPY incubator-singa/tool/mesos/* /root/incubator-singa/tool/mesos/ - -RUN cd /opt && source ./install.sh - -RUN apt-get -y install curl cmake libxml2 libxml2-dev uuid-dev protobuf-compiler libprotobuf-dev libgsasl7-dev libkrb5-dev libboost1.54-all-dev - -RUN cd /opt && source /root/.bashrc && wget -c http://www.eu.apache.org/dist/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz && tar -zxvf hadoop-2.6.0.tar.gz && git clone https://github.com/PivotalRD/libhdfs3.git && cd libhdfs3 && mkdir build && cd build && cmake ../ && make && make install - -#to be removed after SINGA-11 is merged (pulling from Anh's branch instead of pulling from the master) -#RUN source ~/.bashrc && mkdir /root/mesos && cd /root/mesos && git clone https://github.com/ug93tad/incubator-singa && cd incubator-singa && git checkout SINGA-11 && cp -r tool/mesos /root/incubator-singa/tool/ && cd /root/incubator-singa/tool/mesos && make - -# update SINGA, but keep the examples directory -RUN source ~/.bashrc && cd /root/incubator-singa/tool/mesos && make - -COPY incubator-singa/tool/docker/mesos/*.xml /opt/hadoop-2.6.0/etc/hadoop/ -RUN ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa && cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys && echo 'StrictHostKeyChecking no' >> ~/.ssh/config http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/mesos/core-site.xml ---------------------------------------------------------------------- diff --git a/tool/docker/mesos/core-site.xml b/tool/docker/mesos/core-site.xml deleted file mode 100644 index 49a39ca..0000000 --- a/tool/docker/mesos/core-site.xml +++ /dev/null @@ -1,24 +0,0 @@ -<?xml version="1.0" encoding="UTF-8"?> -<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> -<!-- - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. See accompanying LICENSE file. ---> - -<!-- Put site-specific property overrides in this file. --> - -<configuration> - <property> - <name>fs.default.name</name> - <value>hdfs://node0:9000</value> - </property> -</configuration> http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/mesos/hdfs-site.xml ---------------------------------------------------------------------- diff --git a/tool/docker/mesos/hdfs-site.xml b/tool/docker/mesos/hdfs-site.xml deleted file mode 100644 index 2e0f4c6..0000000 --- a/tool/docker/mesos/hdfs-site.xml +++ /dev/null @@ -1,34 +0,0 @@ -<?xml version="1.0" encoding="UTF-8"?> -<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> -<!-- - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. See accompanying LICENSE file. ---> - -<!-- Put site-specific property overrides in this file. --> - -<configuration> -<property> - <name>dfs.replication</name> - <value>1</value> - </property> - -<property> - <name>dfs.name.dir</name> - <value>file:///var/lib/hadoop/hdfs/namenode</value> - </property> - - <property> - <name>dfs.data.dir</name> - <value>file:///var/lib/hadoop/hdfs/datanode</value> - </property> -</configuration> http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/mesos/install.sh ---------------------------------------------------------------------- diff --git a/tool/docker/mesos/install.sh b/tool/docker/mesos/install.sh deleted file mode 100644 index 4b6ae79..0000000 --- a/tool/docker/mesos/install.sh +++ /dev/null @@ -1,35 +0,0 @@ -#!/bin/bash - -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ - -source /root/.bashrc -#download -cd /opt -wget -c http://archive.apache.org/dist/mesos/0.22.0/mesos-0.22.0.tar.gz -wget https://www.comp.nus.edu.sg/~dinhtta/files/mesos_patch -tar -zxvf mesos-0.22.0.tar.gz - -#patch and install mesos -cd /opt/mesos-0.22.0 -patch -p5 < ../mesos_patch -mkdir build; cd build -../configure -make -sudo make install - http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/mesos/mapred-site.xml ---------------------------------------------------------------------- diff --git a/tool/docker/mesos/mapred-site.xml b/tool/docker/mesos/mapred-site.xml deleted file mode 100644 index 42bb81c..0000000 --- a/tool/docker/mesos/mapred-site.xml +++ /dev/null @@ -1,22 +0,0 @@ -<?xml version="1.0"?> -<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> -<!-- - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. See accompanying LICENSE file. ---> - -<!-- Put site-specific property overrides in this file. --> - -<configuration> - <property> - <name>mapreduce.framework.name</name> - <value>yarn</value> - </property> -</configuration> http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/mesos/yarn-site.xml ---------------------------------------------------------------------- diff --git a/tool/docker/mesos/yarn-site.xml b/tool/docker/mesos/yarn-site.xml deleted file mode 100644 index c137af9..0000000 --- a/tool/docker/mesos/yarn-site.xml +++ /dev/null @@ -1,32 +0,0 @@ -<?xml version="1.0"?> -<!-- - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. See accompanying LICENSE file. ---> -<configuration> - -<!-- Site specific YARN configuration properties --> - <property> - <name>yarn.nodemanager.aux-services</name> - <value>mapreduce_shuffle</value> - </property> - - <property> - <name>yarn.resourcemanager.hostname</name> - <value>node0</value> - </property> - - <property> - <name>yarn.resourcemanager.bind-host</name> - <value>0.0.0.0</value> - </property> -</configuration> http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/singa/.bashrc ---------------------------------------------------------------------- diff --git a/tool/docker/singa/.bashrc b/tool/docker/singa/.bashrc deleted file mode 100644 index cf0712f..0000000 --- a/tool/docker/singa/.bashrc +++ /dev/null @@ -1,30 +0,0 @@ -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ - -export LIBRARY_PATH=/opt/OpenBLAS/lib:$LIBRARY_PATH -export LD_LIBRARY_PATH=/opt/OpenBLAS/lib:/usr/local/lib:$LD_LIBRARY_PATH -export CPLUS_INCLUDE_PATH=/opt/OpenBLAS/include:$CPLUS_INCLUDE_PATH -export PATH=$PATH:/opt/jdk1.8.0_60/bin:/opt/bin -export JAVA_HOME=/opt/jdk1.8.0_60 -export SINGA_HOME=/root/incubator-singa -export PATH=$PATH:$SINGA_HOME/bin -alias ls="ls --color=always" -# some more ls aliases -alias ll="ls -alF" -alias la="ls -A" -alias l="ls -CF" http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/singa/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/docker/singa/Dockerfile b/tool/docker/singa/Dockerfile deleted file mode 100644 index 0fc16bc..0000000 --- a/tool/docker/singa/Dockerfile +++ /dev/null @@ -1,34 +0,0 @@ -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ - -#build singa from scratch -FROM ubuntu:14.04 -MAINTAINER Anh Dinh<[email protected]> -RUN rm /bin/sh && ln -s /bin/bash /bin/sh - -RUN apt-get update && apt-get -y install g++-4.8 build-essential git vim wget zip automake libtool man python-dev python-boto libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev openssh-server supervisor - -RUN cd /opt && wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u60-b27/jdk-8u60-linux-x64.tar.gz && cd /opt && tar -zxvf jdk-8u60-linux-x64.tar.gz && rm -rf jdk-8u60-linux-x64.tar.gz - -COPY incubator-singa/tool/docker/singa/ssh.conf /etc/supervisor/conf.d/ -COPY incubator-singa/tool/docker/singa/.bashrc /root/.bashrc -COPY incubator-singa/tool/docker/singa/.vimrc /root/.vimrc -ADD incubator-singa /root/incubator-singa - -RUN cd && source /root/.bashrc && cd incubator-singa && ./thirdparty/install.sh all && cd thirdparty/ && wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz && tar -zxvf protobuf-2.5.0.tar.gz && cd protobuf-2.5.0 && ./configure && make clean && make && make install && cd ../../ && cp thirdparty/install.sh . && rm -rf thirdparty/* && mv install.sh thirdparty/ && ./thirdparty/install.sh zookeeper && ./autogen.sh && ./configure && make && make install && cd examples/cifar10 && mv Makefile.example Makefile && make download && make create && service ssh start - http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/singa/Dockerfile_gpu ---------------------------------------------------------------------- diff --git a/tool/docker/singa/Dockerfile_gpu b/tool/docker/singa/Dockerfile_gpu deleted file mode 100644 index da57479..0000000 --- a/tool/docker/singa/Dockerfile_gpu +++ /dev/null @@ -1,80 +0,0 @@ -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ - -FROM singa/base -MAINTAINER NVIDIA CORPORATION <[email protected]> - -LABEL com.nvidia.volumes.needed="nvidia_driver" - -ENV NVIDIA_GPGKEY_SUM bd841d59a27a406e513db7d405550894188a4c1cd96bf8aa4f82f1b39e0b5c1c -ENV NVIDIA_GPGKEY_FPR 889bee522da690103c4b085ed88c3d385c37d3be - -RUN apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/GPGKEY && \ - apt-key adv --export --no-emit-version -a $NVIDIA_GPGKEY_FPR | tail -n +2 > cudasign.pub && \ - echo "$NVIDIA_GPGKEY_SUM cudasign.pub" | sha256sum -c --strict - && rm cudasign.pub && \ - echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64 /" > /etc/apt/sources.list.d/cuda.list - -ENV CUDA_VERSION 7.0 -LABEL com.nvidia.cuda.version="7.0" - -ENV CUDA_PKG_VERSION 7-0=7.0-28 -RUN apt-get update && apt-get install -y --no-install-recommends --force-yes curl \ - cuda-nvrtc-$CUDA_PKG_VERSION \ - cuda-cusolver-$CUDA_PKG_VERSION \ - cuda-cublas-$CUDA_PKG_VERSION \ - cuda-cufft-$CUDA_PKG_VERSION \ - cuda-curand-$CUDA_PKG_VERSION \ - cuda-cusparse-$CUDA_PKG_VERSION \ - cuda-npp-$CUDA_PKG_VERSION \ - cuda-cudart-$CUDA_PKG_VERSION && \ - ln -s cuda-$CUDA_VERSION /usr/local/cuda - -RUN apt-get install -y --no-install-recommends --force-yes \ - cuda-core-$CUDA_PKG_VERSION \ - cuda-misc-headers-$CUDA_PKG_VERSION \ - cuda-command-line-tools-$CUDA_PKG_VERSION \ - cuda-license-$CUDA_PKG_VERSION \ - cuda-nvrtc-dev-$CUDA_PKG_VERSION \ - cuda-cusolver-dev-$CUDA_PKG_VERSION \ - cuda-cublas-dev-$CUDA_PKG_VERSION \ - cuda-cufft-dev-$CUDA_PKG_VERSION \ - cuda-curand-dev-$CUDA_PKG_VERSION \ - cuda-cusparse-dev-$CUDA_PKG_VERSION \ - cuda-npp-dev-$CUDA_PKG_VERSION \ - cuda-cudart-dev-$CUDA_PKG_VERSION \ - cuda-driver-dev-$CUDA_PKG_VERSION - -RUN echo "/usr/local/cuda/lib" >> /etc/ld.so.conf.d/cuda.conf && \ - echo "/usr/local/cuda/lib64" >> /etc/ld.so.conf.d/cuda.conf && \ - ldconfig - -ENV CUDNN_DOWNLOAD_SUM 98679d5ec039acfd4d81b8bfdc6a6352d6439e921523ff9909d364e706275c2b - -RUN curl -fsSL http://developer.download.nvidia.com/compute/redist/cudnn/v3/cudnn-7.0-linux-x64-v3.0-prod.tgz -O && \ - echo "$CUDNN_DOWNLOAD_SUM cudnn-7.0-linux-x64-v3.0-prod.tgz" | sha256sum -c --strict - && \ - tar -xzf cudnn-7.0-linux-x64-v3.0-prod.tgz -C /usr/local && \ - rm cudnn-7.0-linux-x64-v3.0-prod.tgz && \ - ldconfig - -RUN echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf && \ - echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf - -ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH} -ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64:${LD_LIBRARY_PATH} - -RUN cd && source /root/.bashrc && cd incubator-singa && ./configure --enable-cuda --with-cuda=/usr/local/cuda --enable-cudnn --with-cudnn=/usr/local/cuda && make && make install http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/singa/ssh.conf ---------------------------------------------------------------------- diff --git a/tool/docker/singa/ssh.conf b/tool/docker/singa/ssh.conf deleted file mode 100644 index ac04d7e..0000000 --- a/tool/docker/singa/ssh.conf +++ /dev/null @@ -1,5 +0,0 @@ -[supervisord] -nodaemon=true - -[program:ssh] -command=/usr/sbin/sshd -D http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/README.md ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/README.md b/tool/mesos/docker/README.md new file mode 100644 index 0000000..fd33e7e --- /dev/null +++ b/tool/mesos/docker/README.md @@ -0,0 +1,189 @@ +This guide explains how to set up a development environment for SINGA. A cluster environment can be launched over one or multiple physical nodes. + +1. [Build SINGA base](#build_base) +2. [Build SINGA with Mesos and Hadoop](#build_mesos) +3. [Pre-built images](#pre_built) +4. [Launch and stop SINGA (stand alone mode)](#launch_stand_alone) +5. [Launch pseudo-distributed SINGA on one node](#launch_pseudo) +6. [Launch fully distributed SINGA on multiple nodes](#launch_distributed) + +--- + +<a name="build_base"></a> +#### Build SINGA base image + +```` +$ cd tool/docker/singa +$ sudo docker build -t singa/base . +$ sudo docker images +REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE +singa/base latest XXXX XXX 2.01 GB +```` + +The result is the image containing a built version of SINGA. + +  + + *Figure 1. singa/base Docker image, containing library dependencies and SINGA built from source.* + +-- + +<a name="build_mesos"></a> +#### Build SINGA with Mesos and Hadoop +```` +$ cd tool/docker/mesos +$ sudo docker build -t singa/mesos . +$ sudo docker images +REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE +singa/mesos latest XXXX XXX 4.935 GB +```` +  + + *Figure 2. singa/mesos Docker image, containing Hadoop and Mesos built on +top of SINGA. The default namenode address for Hadoop is `node0:9000`* + +**Notes** A common failure observed during the build process is caused by network failure occuring when downloading dependencies. Simply re-run the build command. + +-- + +<a name="pre_built"></a> +#### Pre-built images on epiC cluster +For users with access to the `epiC` cluster, there are pre-built and loaded Docker images at the following nodes: + + ciidaa-c18 + ciidaa-c19 + +The available images at those nodes are: +```` +REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE +singa/base latest XXXX XXX 2.01 GB +singa/mesos latest XXXX XXX 4.935 GB +weaveworks/weaveexec 1.1.1 XXXX 11 days ago 57.8 MB +weaveworks/weave 1.1.1 XXXX 11 days ago 17.56 MB +```` + +-- + +<a name="launch_stand_alone"></a> +#### Launch and stop SINGA in stand-alone mode +To launch a test environment for a single-node SINGA training, simply start a container from `singa/base` image. The following starts a container called +`XYZ`, then launches a shell in the container: + +```` +$ sudo docker run -dt --name XYZ singa/base /usr/bin/supervisord +$ sudo docker exec -it XYZ /bin/bash +```` + +  + + *Figure 3. Launch SINGA in stand-alone mode: single node training* + +Inside the launched container, the SINGA source directory can be found at `/root/incubator-singa`. + +**Stopping the container** + +```` +$ sudo docker stop XYZ +$ sudo docker rm ZYZ +```` + +-- + +<a name="launch_pseudo"></a> +#### Launch SINGA on pseudo-distributed mode (single node) +To simulate a distributed environment on a single node, one can repeat the +previous step multiple times, each time giving a different name to the +container. Network connections between these containers are already supported, +thus SINGA instances/nodes in these container can readily communicate with each +other. + +The previous approach requires the user to start SINGA instances individually +at each container. Although there's a bash script for that, we provide a better +way. In particular, multiple containers can be started from `singa/mesos` image +which already bundles Mesos and Hadoop with SINGA. Using Mesos makes it easy to +launch, stop and monitor the distributed execution from a single container. +Figure 4 shows `N+1` containers running concurrently at the local host. + +```` +# sudo docker run -dt --name node0 singa/mesos /usr/bin/supervisord +# sudo docker run -dt --name node1 singa/mesos /usr/bin/supervisord +... +```` +  + + *Figure 4. Launch SINGA in pseudo-distributed mode : multiple SINGA nodes over one single machine* + +**Starting SINGA distributed training** + +Refer to the [Mesos +guide](https://github.com/ug93tad/incubator-singa/blob/SINGA-11/tool/mesos/README.md) +for details of how to start training with multiple SINGA instances. + +**Important:** the container that assumes the role of Hadoop's namenode (and often Mesos's and Zookeeper's mater node as well) **must** be named `node0`. Otherwise, the user must log in to individual containers and change the Hadoop configuration separately. + +-- + +<a name="launch_distributed"></a> +#### Launch SINGA on fully distributed mode (multiple nodes) +The previous section has explained how to start a distributed environment on a +single node. But running many containers on one node does not scale. When there +are multiple physical hosts available, it is better to distribute the +containers over them. + +The only extra requirement for the fully distributed mode, as compared with the +pseudo distributed mode, is that the containers from different hosts are able +to transparently communicate with each other. In the pseudo distributed mode, +the local docker engine takes care of such communication. Here, we rely on +[Weave](http://weave.works/guides/weave-docker-ubuntu-simple.html) to make the +communication transparent. The resulting architecture is shown below. + +  + + *Figure 5. Launch SINGA in fully distributed mode: multiple SINGA nodes over multiple machines* + +**Install Weave at all hosts** +``` +$ curl -L git.io/weave -o /usr/local/bin/weave +$ chmod a+x /usr/local/bin/weave +``` + +**Starting Weave** + +Suppose `node0` will be launched at host with IP `111.222.111.222`. + +* At host `111.222.111.222`: +```` +$ weave launch +$ eval "$(weave env)" //if there's error, do `sudo -s` and try again +```` +* At other hosts: +```` +$ weave launch 111.222.111.222 +$ eval "$(weave env)" //if there's error, do `sudo -s` and try again +```` + +**Starting containers** + +The user logs in to each host and starts the container (same as in [pseudo-distributed](#launch_pseudo) mode). Note that container acting as the head node of the cluster must be named `node0` (and be running at the host with IP `111.222.111.222`, for example). + +**_Important_:** when there are other containers sharing the same host as `node0`, say `node1` and `node2` for example, +there're additional changes to be made to `node1` and `node2`. Particularly, log in to each container and edit +`/etc/hosts` file: + +```` +# modified by weave +... +X.Y.Z node0 node0.bridge //<- REMOVE this line +.. +```` +This is to ensure that name resolutions (of `node0`'s address) from `node1` and `node2` are correct. By default, +containers of the same host resolves each other's addresses via the Docker bridge. Instead, we want they to use +addressed given by Weave. + + +**Starting SINGA distributed training** + +Refer to the [Mesos +guide](https://github.com/ug93tad/incubator-singa/blob/SINGA-11/tool/mesos/README.md) +for details of how to start training with multiple SINGA instances. + http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/mesos/.bashrc ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/mesos/.bashrc b/tool/mesos/docker/mesos/.bashrc new file mode 100644 index 0000000..c796a27 --- /dev/null +++ b/tool/mesos/docker/mesos/.bashrc @@ -0,0 +1,32 @@ +#/** +# * Licensed to the Apache Software Foundation (ASF) under one +# * or more contributor license agreements. See the NOTICE file +# * distributed with this work for additional information +# * regarding copyright ownership. The ASF licenses this file +# * to you under the Apache License, Version 2.0 (the +# * "License"); you may not use this file except in compliance +# * with the License. You may obtain a copy of the License at +# * +# * http://www.apache.org/licenses/LICENSE-2.0 +# * +# * Unless required by applicable law or agreed to in writing, software +# * distributed under the License is distributed on an "AS IS" BASIS, +# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# * See the License for the specific language governing permissions and +# * limitations under the License. +# */ + +export LIBRARY_PATH=/opt/OpenBLAS/lib:$LIBRARY_PATH +export LD_LIBRARY_PATH=/opt/OpenBLAS/lib:/usr/local/lib:$LD_LIBRARY_PATH +export CPLUS_INCLUDE_PATH=/opt/OpenBLAS/include:$CPLUS_INCLUDE_PATH +export PATH=/opt/jdk1.8.0_60/bin:/opt/bin:$PATH +export HADOOP_HOME=/opt/hadoop-2.6.0 +export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin +export JAVA_HOME=/opt/jdk1.8.0_60 +alias ls="ls --color=always" +# some more ls aliases +alias ll="ls -alF" +alias la="ls -A" +alias l="ls -CF" +export SINGA_HOME=/root/incubator-singa +export PATH=$PATH:$SINGA_HOME/bin http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/mesos/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/mesos/Dockerfile b/tool/mesos/docker/mesos/Dockerfile new file mode 100644 index 0000000..83142b3 --- /dev/null +++ b/tool/mesos/docker/mesos/Dockerfile @@ -0,0 +1,40 @@ +#/** +# * Licensed to the Apache Software Foundation (ASF) under one +# * or more contributor license agreements. See the NOTICE file +# * distributed with this work for additional information +# * regarding copyright ownership. The ASF licenses this file +# * to you under the Apache License, Version 2.0 (the +# * "License"); you may not use this file except in compliance +# * with the License. You may obtain a copy of the License at +# * +# * http://www.apache.org/licenses/LICENSE-2.0 +# * +# * Unless required by applicable law or agreed to in writing, software +# * distributed under the License is distributed on an "AS IS" BASIS, +# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# * See the License for the specific language governing permissions and +# * limitations under the License. +# */ + +#build Mesos+Hadoop on top of SINGA +FROM singa/base +MAINTAINER Anh Dinh<[email protected]> + +COPY incubator-singa/tool/docker/mesos/.bashrc /root/.bashrc +COPY incubator-singa/tool/docker/mesos/install.sh /opt/install.sh +COPY incubator-singa/tool/mesos/* /root/incubator-singa/tool/mesos/ + +RUN cd /opt && source ./install.sh + +RUN apt-get -y install curl cmake libxml2 libxml2-dev uuid-dev protobuf-compiler libprotobuf-dev libgsasl7-dev libkrb5-dev libboost1.54-all-dev + +RUN cd /opt && source /root/.bashrc && wget -c http://www.eu.apache.org/dist/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz && tar -zxvf hadoop-2.6.0.tar.gz && git clone https://github.com/PivotalRD/libhdfs3.git && cd libhdfs3 && mkdir build && cd build && cmake ../ && make && make install + +#to be removed after SINGA-11 is merged (pulling from Anh's branch instead of pulling from the master) +#RUN source ~/.bashrc && mkdir /root/mesos && cd /root/mesos && git clone https://github.com/ug93tad/incubator-singa && cd incubator-singa && git checkout SINGA-11 && cp -r tool/mesos /root/incubator-singa/tool/ && cd /root/incubator-singa/tool/mesos && make + +# update SINGA, but keep the examples directory +RUN source ~/.bashrc && cd /root/incubator-singa/tool/mesos && make + +COPY incubator-singa/tool/docker/mesos/*.xml /opt/hadoop-2.6.0/etc/hadoop/ +RUN ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa && cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys && echo 'StrictHostKeyChecking no' >> ~/.ssh/config http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/mesos/core-site.xml ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/mesos/core-site.xml b/tool/mesos/docker/mesos/core-site.xml new file mode 100644 index 0000000..49a39ca --- /dev/null +++ b/tool/mesos/docker/mesos/core-site.xml @@ -0,0 +1,24 @@ +<?xml version="1.0" encoding="UTF-8"?> +<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +<!-- Put site-specific property overrides in this file. --> + +<configuration> + <property> + <name>fs.default.name</name> + <value>hdfs://node0:9000</value> + </property> +</configuration> http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/mesos/hdfs-site.xml ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/mesos/hdfs-site.xml b/tool/mesos/docker/mesos/hdfs-site.xml new file mode 100644 index 0000000..2e0f4c6 --- /dev/null +++ b/tool/mesos/docker/mesos/hdfs-site.xml @@ -0,0 +1,34 @@ +<?xml version="1.0" encoding="UTF-8"?> +<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +<!-- Put site-specific property overrides in this file. --> + +<configuration> +<property> + <name>dfs.replication</name> + <value>1</value> + </property> + +<property> + <name>dfs.name.dir</name> + <value>file:///var/lib/hadoop/hdfs/namenode</value> + </property> + + <property> + <name>dfs.data.dir</name> + <value>file:///var/lib/hadoop/hdfs/datanode</value> + </property> +</configuration> http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/mesos/install.sh ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/mesos/install.sh b/tool/mesos/docker/mesos/install.sh new file mode 100644 index 0000000..4b6ae79 --- /dev/null +++ b/tool/mesos/docker/mesos/install.sh @@ -0,0 +1,35 @@ +#!/bin/bash + +#/** +# * Licensed to the Apache Software Foundation (ASF) under one +# * or more contributor license agreements. See the NOTICE file +# * distributed with this work for additional information +# * regarding copyright ownership. The ASF licenses this file +# * to you under the Apache License, Version 2.0 (the +# * "License"); you may not use this file except in compliance +# * with the License. You may obtain a copy of the License at +# * +# * http://www.apache.org/licenses/LICENSE-2.0 +# * +# * Unless required by applicable law or agreed to in writing, software +# * distributed under the License is distributed on an "AS IS" BASIS, +# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# * See the License for the specific language governing permissions and +# * limitations under the License. +# */ + +source /root/.bashrc +#download +cd /opt +wget -c http://archive.apache.org/dist/mesos/0.22.0/mesos-0.22.0.tar.gz +wget https://www.comp.nus.edu.sg/~dinhtta/files/mesos_patch +tar -zxvf mesos-0.22.0.tar.gz + +#patch and install mesos +cd /opt/mesos-0.22.0 +patch -p5 < ../mesos_patch +mkdir build; cd build +../configure +make +sudo make install + http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/mesos/mapred-site.xml ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/mesos/mapred-site.xml b/tool/mesos/docker/mesos/mapred-site.xml new file mode 100644 index 0000000..42bb81c --- /dev/null +++ b/tool/mesos/docker/mesos/mapred-site.xml @@ -0,0 +1,22 @@ +<?xml version="1.0"?> +<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +<!-- Put site-specific property overrides in this file. --> + +<configuration> + <property> + <name>mapreduce.framework.name</name> + <value>yarn</value> + </property> +</configuration> http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/mesos/yarn-site.xml ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/mesos/yarn-site.xml b/tool/mesos/docker/mesos/yarn-site.xml new file mode 100644 index 0000000..c137af9 --- /dev/null +++ b/tool/mesos/docker/mesos/yarn-site.xml @@ -0,0 +1,32 @@ +<?xml version="1.0"?> +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> +<configuration> + +<!-- Site specific YARN configuration properties --> + <property> + <name>yarn.nodemanager.aux-services</name> + <value>mapreduce_shuffle</value> + </property> + + <property> + <name>yarn.resourcemanager.hostname</name> + <value>node0</value> + </property> + + <property> + <name>yarn.resourcemanager.bind-host</name> + <value>0.0.0.0</value> + </property> +</configuration> http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/singa/.bashrc ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/singa/.bashrc b/tool/mesos/docker/singa/.bashrc new file mode 100644 index 0000000..cf0712f --- /dev/null +++ b/tool/mesos/docker/singa/.bashrc @@ -0,0 +1,30 @@ +#/** +# * Licensed to the Apache Software Foundation (ASF) under one +# * or more contributor license agreements. See the NOTICE file +# * distributed with this work for additional information +# * regarding copyright ownership. The ASF licenses this file +# * to you under the Apache License, Version 2.0 (the +# * "License"); you may not use this file except in compliance +# * with the License. You may obtain a copy of the License at +# * +# * http://www.apache.org/licenses/LICENSE-2.0 +# * +# * Unless required by applicable law or agreed to in writing, software +# * distributed under the License is distributed on an "AS IS" BASIS, +# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# * See the License for the specific language governing permissions and +# * limitations under the License. +# */ + +export LIBRARY_PATH=/opt/OpenBLAS/lib:$LIBRARY_PATH +export LD_LIBRARY_PATH=/opt/OpenBLAS/lib:/usr/local/lib:$LD_LIBRARY_PATH +export CPLUS_INCLUDE_PATH=/opt/OpenBLAS/include:$CPLUS_INCLUDE_PATH +export PATH=$PATH:/opt/jdk1.8.0_60/bin:/opt/bin +export JAVA_HOME=/opt/jdk1.8.0_60 +export SINGA_HOME=/root/incubator-singa +export PATH=$PATH:$SINGA_HOME/bin +alias ls="ls --color=always" +# some more ls aliases +alias ll="ls -alF" +alias la="ls -A" +alias l="ls -CF" http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/singa/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/singa/Dockerfile b/tool/mesos/docker/singa/Dockerfile new file mode 100644 index 0000000..0fc16bc --- /dev/null +++ b/tool/mesos/docker/singa/Dockerfile @@ -0,0 +1,34 @@ +#/** +# * Licensed to the Apache Software Foundation (ASF) under one +# * or more contributor license agreements. See the NOTICE file +# * distributed with this work for additional information +# * regarding copyright ownership. The ASF licenses this file +# * to you under the Apache License, Version 2.0 (the +# * "License"); you may not use this file except in compliance +# * with the License. You may obtain a copy of the License at +# * +# * http://www.apache.org/licenses/LICENSE-2.0 +# * +# * Unless required by applicable law or agreed to in writing, software +# * distributed under the License is distributed on an "AS IS" BASIS, +# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# * See the License for the specific language governing permissions and +# * limitations under the License. +# */ + +#build singa from scratch +FROM ubuntu:14.04 +MAINTAINER Anh Dinh<[email protected]> +RUN rm /bin/sh && ln -s /bin/bash /bin/sh + +RUN apt-get update && apt-get -y install g++-4.8 build-essential git vim wget zip automake libtool man python-dev python-boto libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev openssh-server supervisor + +RUN cd /opt && wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u60-b27/jdk-8u60-linux-x64.tar.gz && cd /opt && tar -zxvf jdk-8u60-linux-x64.tar.gz && rm -rf jdk-8u60-linux-x64.tar.gz + +COPY incubator-singa/tool/docker/singa/ssh.conf /etc/supervisor/conf.d/ +COPY incubator-singa/tool/docker/singa/.bashrc /root/.bashrc +COPY incubator-singa/tool/docker/singa/.vimrc /root/.vimrc +ADD incubator-singa /root/incubator-singa + +RUN cd && source /root/.bashrc && cd incubator-singa && ./thirdparty/install.sh all && cd thirdparty/ && wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz && tar -zxvf protobuf-2.5.0.tar.gz && cd protobuf-2.5.0 && ./configure && make clean && make && make install && cd ../../ && cp thirdparty/install.sh . && rm -rf thirdparty/* && mv install.sh thirdparty/ && ./thirdparty/install.sh zookeeper && ./autogen.sh && ./configure && make && make install && cd examples/cifar10 && mv Makefile.example Makefile && make download && make create && service ssh start + http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/singa/Dockerfile_gpu ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/singa/Dockerfile_gpu b/tool/mesos/docker/singa/Dockerfile_gpu new file mode 100644 index 0000000..da57479 --- /dev/null +++ b/tool/mesos/docker/singa/Dockerfile_gpu @@ -0,0 +1,80 @@ +#/** +# * Licensed to the Apache Software Foundation (ASF) under one +# * or more contributor license agreements. See the NOTICE file +# * distributed with this work for additional information +# * regarding copyright ownership. The ASF licenses this file +# * to you under the Apache License, Version 2.0 (the +# * "License"); you may not use this file except in compliance +# * with the License. You may obtain a copy of the License at +# * +# * http://www.apache.org/licenses/LICENSE-2.0 +# * +# * Unless required by applicable law or agreed to in writing, software +# * distributed under the License is distributed on an "AS IS" BASIS, +# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# * See the License for the specific language governing permissions and +# * limitations under the License. +# */ + +FROM singa/base +MAINTAINER NVIDIA CORPORATION <[email protected]> + +LABEL com.nvidia.volumes.needed="nvidia_driver" + +ENV NVIDIA_GPGKEY_SUM bd841d59a27a406e513db7d405550894188a4c1cd96bf8aa4f82f1b39e0b5c1c +ENV NVIDIA_GPGKEY_FPR 889bee522da690103c4b085ed88c3d385c37d3be + +RUN apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/GPGKEY && \ + apt-key adv --export --no-emit-version -a $NVIDIA_GPGKEY_FPR | tail -n +2 > cudasign.pub && \ + echo "$NVIDIA_GPGKEY_SUM cudasign.pub" | sha256sum -c --strict - && rm cudasign.pub && \ + echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64 /" > /etc/apt/sources.list.d/cuda.list + +ENV CUDA_VERSION 7.0 +LABEL com.nvidia.cuda.version="7.0" + +ENV CUDA_PKG_VERSION 7-0=7.0-28 +RUN apt-get update && apt-get install -y --no-install-recommends --force-yes curl \ + cuda-nvrtc-$CUDA_PKG_VERSION \ + cuda-cusolver-$CUDA_PKG_VERSION \ + cuda-cublas-$CUDA_PKG_VERSION \ + cuda-cufft-$CUDA_PKG_VERSION \ + cuda-curand-$CUDA_PKG_VERSION \ + cuda-cusparse-$CUDA_PKG_VERSION \ + cuda-npp-$CUDA_PKG_VERSION \ + cuda-cudart-$CUDA_PKG_VERSION && \ + ln -s cuda-$CUDA_VERSION /usr/local/cuda + +RUN apt-get install -y --no-install-recommends --force-yes \ + cuda-core-$CUDA_PKG_VERSION \ + cuda-misc-headers-$CUDA_PKG_VERSION \ + cuda-command-line-tools-$CUDA_PKG_VERSION \ + cuda-license-$CUDA_PKG_VERSION \ + cuda-nvrtc-dev-$CUDA_PKG_VERSION \ + cuda-cusolver-dev-$CUDA_PKG_VERSION \ + cuda-cublas-dev-$CUDA_PKG_VERSION \ + cuda-cufft-dev-$CUDA_PKG_VERSION \ + cuda-curand-dev-$CUDA_PKG_VERSION \ + cuda-cusparse-dev-$CUDA_PKG_VERSION \ + cuda-npp-dev-$CUDA_PKG_VERSION \ + cuda-cudart-dev-$CUDA_PKG_VERSION \ + cuda-driver-dev-$CUDA_PKG_VERSION + +RUN echo "/usr/local/cuda/lib" >> /etc/ld.so.conf.d/cuda.conf && \ + echo "/usr/local/cuda/lib64" >> /etc/ld.so.conf.d/cuda.conf && \ + ldconfig + +ENV CUDNN_DOWNLOAD_SUM 98679d5ec039acfd4d81b8bfdc6a6352d6439e921523ff9909d364e706275c2b + +RUN curl -fsSL http://developer.download.nvidia.com/compute/redist/cudnn/v3/cudnn-7.0-linux-x64-v3.0-prod.tgz -O && \ + echo "$CUDNN_DOWNLOAD_SUM cudnn-7.0-linux-x64-v3.0-prod.tgz" | sha256sum -c --strict - && \ + tar -xzf cudnn-7.0-linux-x64-v3.0-prod.tgz -C /usr/local && \ + rm cudnn-7.0-linux-x64-v3.0-prod.tgz && \ + ldconfig + +RUN echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf && \ + echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf + +ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH} +ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64:${LD_LIBRARY_PATH} + +RUN cd && source /root/.bashrc && cd incubator-singa && ./configure --enable-cuda --with-cuda=/usr/local/cuda --enable-cudnn --with-cudnn=/usr/local/cuda && make && make install http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/singa/ssh.conf ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/singa/ssh.conf b/tool/mesos/docker/singa/ssh.conf new file mode 100644 index 0000000..ac04d7e --- /dev/null +++ b/tool/mesos/docker/singa/ssh.conf @@ -0,0 +1,5 @@ +[supervisord] +nodaemon=true + +[program:ssh] +command=/usr/sbin/sshd -D
