preparing for v1.1-rc1: remove outdated files; fix links
Project: http://git-wip-us.apache.org/repos/asf/incubator-singa/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-singa/commit/b2e21c62 Tree: http://git-wip-us.apache.org/repos/asf/incubator-singa/tree/b2e21c62 Diff: http://git-wip-us.apache.org/repos/asf/incubator-singa/diff/b2e21c62 Branch: refs/heads/master Commit: b2e21c626fa6983a6f9166b7ba426a5e004b2073 Parents: 5a6c303 Author: wang wei <[email protected]> Authored: Mon Jan 23 23:25:16 2017 +0800 Committer: Wei Wang <[email protected]> Committed: Tue Jan 24 16:55:06 2017 +0800 ---------------------------------------------------------------------- CONTRIBUTING.md | 2 +- NOTICE | 2 +- RELEASE_NOTES | 32 +- bin/singa-cleanup.sh | 32 -- bin/singa-console.sh | 85 ---- bin/singa-env.sh | 68 --- bin/singa-run.sh | 107 ---- bin/singa-stop.sh | 47 -- bin/zk-service.sh | 72 --- doc/en/develop/how-contribute.md | 10 +- doc/en/develop/schedule.rst | 81 +-- rat-excludes | 3 +- thirdparty/install.sh | 714 --------------------------- tool/graph.py | 40 -- tool/mesos/Makefile | 25 - tool/mesos/README.md | 86 ---- tool/mesos/docker/README.md | 189 ------- tool/mesos/docker/mesos/.bashrc | 32 -- tool/mesos/docker/mesos/Dockerfile | 40 -- tool/mesos/docker/mesos/core-site.xml | 24 - tool/mesos/docker/mesos/hdfs-site.xml | 34 -- tool/mesos/docker/mesos/install.sh | 35 -- tool/mesos/docker/mesos/mapred-site.xml | 22 - tool/mesos/docker/mesos/yarn-site.xml | 32 -- tool/mesos/docker/singa/.bashrc | 30 -- tool/mesos/docker/singa/Dockerfile | 34 -- tool/mesos/docker/singa/Dockerfile_gpu | 80 --- tool/mesos/docker/singa/ssh.conf | 5 - tool/mesos/scheduler.conf | 3 - tool/mesos/scheduler.proto | 33 -- tool/mesos/singa_scheduler.cc | 426 ---------------- tool/node.sh | 88 ---- 32 files changed, 66 insertions(+), 2447 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/CONTRIBUTING.md ---------------------------------------------------------------------- diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 442f477..6fd833c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,6 +4,6 @@ [How to contribute code](./doc/en/develop/contribute-code.md) -[How to contribute documentation](./doc/en/develop/contribute-docs.md) +[How to contribute documentation](./doc/README.md) http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/NOTICE ---------------------------------------------------------------------- diff --git a/NOTICE b/NOTICE index 092ec36..6fc258b 100644 --- a/NOTICE +++ b/NOTICE @@ -1,5 +1,5 @@ Apache SINGA -Copyright 2016 The Apache Software Foundation +Copyright 2017 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/RELEASE_NOTES ---------------------------------------------------------------------- diff --git a/RELEASE_NOTES b/RELEASE_NOTES index d9970e3..9cafffe 100644 --- a/RELEASE_NOTES +++ b/RELEASE_NOTES @@ -8,36 +8,36 @@ This release includes following features: * [SINGA-296] Add sign and to_host function for pysinga tensor module * Model components + * [SINGA-254] Implement Adam for V1 * [SINGA-264] Extend the FeedForwardNet to accept multiple inputs - * [SINGA-278] Convert trained caffe parameters to singa - * [SINGA-287] Add memory size check for cudnn convolution - * [SINGA-275] Add Cross Entropy Loss for multiple labels * [SINGA-267] Add spatial mode in batch normalization layer - * [SINGA-254] Implement Adam for V1 * [SINGA-271] Add Concat and Slice layers + * [SINGA-275] Add Cross Entropy Loss for multiple labels + * [SINGA-278] Convert trained caffe parameters to singa + * [SINGA-287] Add memory size check for cudnn convolution * Utility functions and CI - * [SINGA-261] Add version ID into the checkpoint files + * [SINGA-242] Compile all source files into a single library. + * [SINGA-244] Separating swig interface and python binding files + * [SINGA-246] Imgtool for image augmentation + * [SINGA-247] Add windows support for singa * [SINGA-251] Implement image loader for pysinga - * [SINGA-284] Add python unittest into Jenkins and link static libs into whl file - * [SINGA-280] Jenkins CI support - * [SINGA-288] Publish wheel of PySINGA generated by Jenkins to public servers - * [SINGA-266] Add Rafiki python toolkits + * [SINGA-252] Use the snapshot methods to dump and load models for pysinga * [SINGA-255] Compile mandatory depedent libaries together with SINGA code - * [SINGA-247] Add windows support for singa * [SINGA-259] Add maven pom file for building java classes - * [SINGA-246] Imgtool for image augmentation - * [SINGA-252] Use the snapshot methods to dump and load models for pysinga - * [SINGA-242] Compile all source files into a single library. - * [SINGA-251] Implement image loader for pysinga - * [SINGA-244] Separating swig interface and python binding files + * [SINGA-261] Add version ID into the checkpoint files + * [SINGA-266] Add Rafiki python toolkits * [SINGA-273] Improve license and contributions + * [SINGA-284] Add python unittest into Jenkins and link static libs into whl file + * [SINGA-280] Jenkins CI support + * [SINGA-288] Publish wheel of PySINGA generated by Jenkins to public servers * Documentation and usability - * [SINGA-268] Add IPython notebooks to the documentation * [SINGA-263] Create Amazon Machine Image + * [SINGA-268] Add IPython notebooks to the documentation * [SINGA-276] Create docker images * [SINGA-289] Update SINGA website automatically using Jenkins + * [SINGA-295] Add an example of image classification using GoogleNet * Bugs fixed * [SINGA-245] float as the first operand can not multiply with a tensor object http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/bin/singa-cleanup.sh ---------------------------------------------------------------------- diff --git a/bin/singa-cleanup.sh b/bin/singa-cleanup.sh deleted file mode 100755 index 88b947d..0000000 --- a/bin/singa-cleanup.sh +++ /dev/null @@ -1,32 +0,0 @@ -#!/usr/bin/env bash -# -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ -# -# kill all singa jobs and clean up zookeeper -# - -# get environment variables -. `dirname "${BASH_SOURCE-$0}"`/singa-env.sh -cd $SINGA_HOME - -# kill singa jobs -$SINGA_BIN/singa-stop.sh || exit 1 - -# cleanup whole zookeeper -./singatool cleanup || exit 1 http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/bin/singa-console.sh ---------------------------------------------------------------------- diff --git a/bin/singa-console.sh b/bin/singa-console.sh deleted file mode 100755 index 06ee945..0000000 --- a/bin/singa-console.sh +++ /dev/null @@ -1,85 +0,0 @@ -#!/usr/bin/env bash -# -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ -# -# console to list/view/kill singa jobs -# - -usage="Usage: singa-console.sh <command> <args>\n - list : list running singa jobs\n - view <job id> : view procs of a singa job\n - kill <job id> : kill a singa job" - -if [ $# == 0 ]; then - echo -e $usage - exit 1 -fi - -# get environment variables -. `dirname "${BASH_SOURCE-$0}"`/singa-env.sh -cd $SINGA_HOME - -case $1 in - list) - if [ $# != 1 ]; then - echo -e $usage - exit 1 - fi - ./singatool list || exit 1 - ;; - - view) - if [ $# != 2 ]; then - echo -e $usage - exit 1 - fi - ./singatool view $2 || exit 1 - ;; - - kill) - if [ $# != 2 ]; then - echo -e $usage - exit 1 - fi - hosts=`./singatool view "$2"` - [ $? == 0 ] || exit 1 - ssh_options="-oStrictHostKeyChecking=no \ - -oUserKnownHostsFile=/dev/null \ - -oLogLevel=quiet" - if [ `head -1 "$SINGA_CONF"/hostfile` == localhost ]; then - local_procs=1 - fi - for i in ${hosts[@]}; do - proc=(`echo $i | tr '|' ' '`) - singa_kill="kill -9 "${proc[1]} - if [ -z $local_procs ]; then - echo Kill singa @ $i ... - ssh $ssh_options ${proc[0]} $singa_kill - else - echo Kill singa @ ${proc[1]} ... - $singa_kill - fi - done - ./singatool remove $2 || exit 1 - ;; - - *) - echo -e $usage - exit 1 -esac http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/bin/singa-env.sh ---------------------------------------------------------------------- diff --git a/bin/singa-env.sh b/bin/singa-env.sh deleted file mode 100755 index 8adeade..0000000 --- a/bin/singa-env.sh +++ /dev/null @@ -1,68 +0,0 @@ -#!/usr/bin/env bash -# -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ -# -# set singa environment variables, includes: -# * SINGA_HOME -# * SINGA_BIN -# * SINGA_CONF -# * SINGA_LOG -# * ZK_HOME -# * SINGA_MANAGES_ZK -# - -# exit if varaiables already set -[ -z $SINGA_ENV_DONE ] || exit 0 - -# set SINGA_BIN -if [ -z $SINGA_BIN ]; then - SINGA_BIN=`dirname "${BASH_SOURCE-$0}"` - SINGA_BIN=`cd "$SINGA_BIN">/dev/null; pwd` -fi - -# set SINGA_HOME -if [ -z $SINGA_HOME ]; then - SINGA_HOME=`cd "$SINGA_BIN/..">/dev/null; pwd` -fi - -# set SINGA_CONF -if [ -z $SINGA_CONF ]; then - SINGA_CONF=$SINGA_HOME/conf -fi - -# set SINGA_LOG -if [ -z $SINGA_LOG ]; then - # add -confdir arg, so no need to run under SINGA_HOME - SINGA_LOG=`"$SINGA_HOME"/singatool getlogdir -confdir "$SINGA_CONF"` - [ $? == 0 ] || exit 1 -fi - -# set ZK_HOME -if [ -z $ZK_HOME ]; then - ZK_HOME=$SINGA_HOME/thirdparty/zookeeper-3.4.6 - SINGA_MANAGES_ZK=true -fi - -# set SINGA_MANAGES_ZK -if [ -z $SINGA_MANAGES_ZK ]; then - SINGA_MANAGES_ZK=false -fi - -# mark that we have done all -SINGA_ENV_DONE=1 http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/bin/singa-run.sh ---------------------------------------------------------------------- diff --git a/bin/singa-run.sh b/bin/singa-run.sh deleted file mode 100755 index 763130d..0000000 --- a/bin/singa-run.sh +++ /dev/null @@ -1,107 +0,0 @@ -#!/usr/bin/env bash -# -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ -# -# run a singa job -# - -usage="Usage: singa-run.sh [ arguments ]\n - -exec <binary or python script> : if want to use own singa driver\n - -conf <job config file> : need cluster conf if train in a cluster - -resume : if want to recover a job" - -# parse arguments -# extract and remove '-exec' and '-conf' -# other arguments remain untouched -exe=./singa -while [ $# != 0 ]; do - if [ $1 == "-exec" ]; then - shift - exe=$1 - elif [ $1 == "-conf" ]; then - shift - conf=$1 - else - args="$args $1" - fi - shift -done - -# get environment variables -. `dirname "${BASH_SOURCE-$0}"`/singa-env.sh - -# change conf to an absolute path -if [ ! -z $conf ]; then - conf_dir=`dirname "$conf"` - conf_dir=`cd "$conf_dir">/dev/null; pwd` - conf_base=`basename "$conf"` - job_conf=$conf_dir/$conf_base - if [ ! -f $job_conf ]; then - echo $job_conf not exists - exit 1 - fi -fi - -# go to singa home to execute binary -cd $SINGA_HOME - -# generate unique job id -job_id=`./singatool create` -[ $? == 0 ] || exit 1 -echo Unique JOB_ID is $job_id - -# generate job info dir -# format: job-JOB_ID-YYYYMMDD-HHMMSS -log_dir=$SINGA_LOG/job-info/job-$job_id-$(date '+%Y%m%d-%H%M%S') -mkdir -p $log_dir -echo Record job information to $log_dir - -# generate host file -host_file=$log_dir/job.hosts -./singatool genhost $job_conf 1>$host_file || exit 1 - -# set command to run singa -singa_run="$exe $args \ - -singa_conf $SINGA_HOME/conf/singa.conf \ - -singa_job $job_id" -# add -conf if exists -if [ ! -z $job_conf ]; then - singa_run="$singa_run -conf $job_conf" -fi -singa_sshrun="cd $SINGA_HOME; source $SINGA_HOME/conf/profile; $singa_run" - -# ssh and start singa processes -ssh_options="-oStrictHostKeyChecking=no \ --oUserKnownHostsFile=/dev/null \ --oLogLevel=quiet" -hosts=`cat $host_file | cut -d ' ' -f 1` -for i in ${hosts[@]} ; do - if [ $i = localhost ] ; then - echo Executing : $singa_run - $singa_run & - else - echo Executing @ $i : $singa_sshrun - ssh $ssh_options $i $singa_sshrun " -host " $i & - fi -done - -# generate pid list for this job -sleep 2 -./singatool view $job_id 1>$log_dir/job.pids || exit 1 -wait http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/bin/singa-stop.sh ---------------------------------------------------------------------- diff --git a/bin/singa-stop.sh b/bin/singa-stop.sh deleted file mode 100755 index 446eaf8..0000000 --- a/bin/singa-stop.sh +++ /dev/null @@ -1,47 +0,0 @@ -#!/usr/bin/env bash -# -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ -# -# kill all singa jobs -# - -# get environment variables -. `dirname "${BASH_SOURCE-$0}"`/singa-env.sh -cd $SINGA_HOME - -# kill singa processes -host_file=$SINGA_CONF/hostfile -ssh_options="-oStrictHostKeyChecking=no \ - -oUserKnownHostsFile=/dev/null \ - -oLogLevel=quiet" -hosts=`cat $host_file | cut -d ' ' -f 1` -singa_kill="killall -q -s SIGKILL -r singa" -for i in ${hosts[@]}; do - echo Kill singa @ $i ... - if [ $i == localhost ]; then - $singa_kill - else - ssh $ssh_options $i $singa_kill - fi -done -# wait for killall command -sleep 2 - -# remove job paths in zookeeper -./singatool removeall || exit 1 http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/bin/zk-service.sh ---------------------------------------------------------------------- diff --git a/bin/zk-service.sh b/bin/zk-service.sh deleted file mode 100755 index e7bb67b..0000000 --- a/bin/zk-service.sh +++ /dev/null @@ -1,72 +0,0 @@ -#!/usr/bin/env bash -# -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ -# -# manage ZooKeeper service -# - -usage="Usage: zk-service.sh [start|stop]" - -if [ $# != 1 ]; then - echo $usage - exit 1 -fi - -# get environment variables -. `dirname "${BASH_SOURCE-$0}"`/singa-env.sh - -# check if singa manages zookeeper service -if [ $SINGA_MANAGES_ZK != true ]; then - echo "Singa does not manage a valid zookeeper service (SINGA_MANAGES_ZK != true)" - exit 1 -fi - -# check zookeeper installation -if [ ! -d $ZK_HOME ]; then - echo "zookeeper not found at $ZK_HOME" - echo "if you do not have zookeeper service, please install:" - echo " $SINGA_HOME/thirdparty/install.sh zookeeper" - echo "otherwise, please set ZK_HOME correctly" - exit 1 -fi - -# get command -case $1 in - start) - # start zk service - # check zoo.cfg - if [ ! -f $ZK_HOME/conf/zoo.cfg ]; then - echo "zoo.cfg not found, create from sample.cfg" - cp $ZK_HOME/conf/zoo_sample.cfg $ZK_HOME/conf/zoo.cfg - fi - # cd to SINGA_HOME as zookeeper.out will be here - cd $SINGA_HOME - $ZK_HOME/bin/zkServer.sh start - ;; - - stop) - # stop zk service - $ZK_HOME/bin/zkServer.sh stop - ;; - - *) - echo $usage - exit 1 -esac - http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/doc/en/develop/how-contribute.md ---------------------------------------------------------------------- diff --git a/doc/en/develop/how-contribute.md b/doc/en/develop/how-contribute.md index 8e8e5c4..bfc4cec 100644 --- a/doc/en/develop/how-contribute.md +++ b/doc/en/develop/how-contribute.md @@ -2,8 +2,8 @@ As with any open source project, there are several ways you can help: -* Join the [mailing list](../community/mail-lists.html) and answer other user's questions. -* [Build Singa](../docs/installation.html) by yourself. -* Report bugs, feature requests and other issues in the [issue tracking](../community/issue-tracking.html) application. -* Check SINGA's [development schedule](schedule.html) and [contribute code](contribute-code.html) by providing patches. -* [Help with the documentation](contribute-docs.html) by updating webpages that are lacking or unclear. +* Join the [mailing list](http://singa.apache.org/en/community/mail-lists.html) and answer other user's questions. +* [Build Singa](http://singa.apache.org/en/docs/installation.html) by yourself. +* Report bugs, feature requests and other issues in the [issue tracking](http://singa.apache.org/en/community/issue-tracking.html) application. +* Check SINGA's [development schedule](http://singa.apache.org/en/develop/schedule.html) and [contribute code](http://singa.apache.org/en/develop/contribute-code.html) by providing patches. +* [Help with the documentation](http://singa.apache.org/en/develop/contribute-docs.html) by updating webpages that are lacking or unclear. http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/doc/en/develop/schedule.rst ---------------------------------------------------------------------- diff --git a/doc/en/develop/schedule.rst b/doc/en/develop/schedule.rst index c097407..0eb4f90 100644 --- a/doc/en/develop/schedule.rst +++ b/doc/en/develop/schedule.rst @@ -22,42 +22,45 @@ Development Schedule .. csv-table:: :header: "Release","Module","Feature" - "0.1 Sep 2015 ","Neural Network ","Feed forward neural network, including CNN, MLP " - " "," ","RBM-like model, including RBM " - " "," ","Recurrent neural network, including standard RNN " - " ","Architecture ","One worker group on single node (with data partition) " - " "," ","Multi worker groups on single node using `Hogwild <http://www.eecs.berkeley.edu/~brecht/papers/hogwildTR.pdf>`_ " - " "," ","Distributed Hogwild" - " "," ","Multi groups across nodes, like `Downpour <http://papers.nips.cc/paper/4687-large-scale-distributed-deep-networks>`_" - " "," ","All-Reduce training architecture like `DeepImage <http://arxiv.org/abs/1501.02876>`_ " - " "," ","Load-balance among servers " - " ","Failure recovery ","Checkpoint and restore " - " ","Tools ","Installation with GNU auto Tools " - "0.2 Jan 2016 ","Neural Network ","Feed forward neural network, including AlexNet, cuDNN layers,Tools " - " "," ","Recurrent neural network, including GRULayer and BPTT " - " "," ","Model partition and hybrid partition " - " ","Tools ","Integration with Mesos for resource management " - " "," ","Prepare Docker images for deployment" - " "," ","Visualization of neural net and debug information " - " ","Binding ","Python binding for major components " - " ","GPU ","Single node with multiple GPUs " - "0.3 April 2016 ","GPU ","Multiple nodes, each with multiple GPUs" - " "," ","Heterogeneous training using both GPU and CPU `CcT <http://arxiv.org/abs/1504.04343>`_" - " "," ","Support cuDNN v4 " - " ","Installation ","Remove dependency on ZeroMQ, CZMQ, Zookeeper for single node training" - " ","Updater ","Add new SGD updaters including Adam, AdamMax and AdaDelta" - " ","Binding ","Enhance Python binding for training" - "1.0 Sep 2016 ","Programming abstraction ","Tensor with linear algebra, neural net and random operations " - " "," ","Updater for distributed parameter updating " - " ","Hardware ","Use Cuda and Cudnn for Nvidia GPU" - " "," ","Use OpenCL for AMD GPU or other devices" - " ","Cross-platform ","To extend from Linux to MacOS" - " "," ","Large image models, e.g., `VGG <https://arxiv.org/pdf/1409.1556.pdf>`_ and `Residual Net <http://arxiv.org/abs/1512.03385>`_" - "1.1 Dec 2016 ","Model Zoo ","Health-care models and popular image models" - " ","Caffe converter ","Use SINGA to train models configured in caffe proto files" - " ","Memory optimization ","Replace CNMEM with new memory pool to reduce memory footprint" - " ","Distributed training ","Migrate distributed training frameworks from V0.3" - " ","Compilation and installation ","Windows suppport" - " "," ","Simplify the installation by compiling protobuf and openblas together with SINGA" - " "," ","Build python wheel automatically using Jenkins" - " "," ","Deploy SINGA programs on Android phones for prediction tasks" + "v0.1 Sep 2015 ","Neural Network ","Feed forward neural network, including CNN, MLP " + " "," ","RBM-like model, including RBM " + " "," ","Recurrent neural network, including standard RNN " + " ","Architecture ","One worker group on single node (with data partition) " + " "," ","Multi worker groups on single node using `Hogwild <http://www.eecs.berkeley.edu/~brecht/papers/hogwildTR.pdf>`_ " + " "," ","Distributed Hogwild" + " "," ","Multi groups across nodes, like `Downpour <http://papers.nips.cc/paper/4687-large-scale-distributed-deep-networks>`_" + " "," ","All-Reduce training architecture like `DeepImage <http://arxiv.org/abs/1501.02876>`_ " + " "," ","Load-balance among servers " + " ","Failure recovery ","Checkpoint and restore " + " ","Tools ","Installation with GNU auto Tools " + "v0.2 Jan 2016 ","Neural Network ","Feed forward neural network, including AlexNet, cuDNN layers,Tools " + " "," ","Recurrent neural network, including GRULayer and BPTT " + " "," ","Model partition and hybrid partition " + " ","Tools ","Integration with Mesos for resource management " + " "," ","Prepare Docker images for deployment" + " "," ","Visualization of neural net and debug information " + " ","Binding ","Python binding for major components " + " ","GPU ","Single node with multiple GPUs " + "v0.3 April 2016 ","GPU ","Multiple nodes, each with multiple GPUs" + " "," ","Heterogeneous training using both GPU and CPU `CcT <http://arxiv.org/abs/1504.04343>`_" + " "," ","Support cuDNN v4 " + " ","Installation ","Remove dependency on ZeroMQ, CZMQ, Zookeeper for single node training" + " ","Updater ","Add new SGD updaters including Adam, AdamMax and AdaDelta" + " ","Binding ","Enhance Python binding for training" + "v1.0 Sep 2016 ","Programming abstraction ","Tensor with linear algebra, neural net and random operations " + " "," ","Updater for distributed parameter updating " + " ","Hardware ","Use Cuda and Cudnn for Nvidia GPU" + " "," ","Use OpenCL for AMD GPU or other devices" + " ","Cross-platform ","To extend from Linux to MacOS" + " "," ","Large image models, e.g., `VGG <https://arxiv.org/pdf/1409.1556.pdf>`_ and `Residual Net <http://arxiv.org/abs/1512.03385>`_" + "v1.1 Jan 2017 ","Model Zoo ","GoogleNet; Health-care models" + " ","Caffe converter ","Use SINGA to train models configured in caffe proto files" + " ","Model components ","Add concat and slice layers; accept multiple inputs to the net" + " ","Compilation and installation ","Windows suppport" + " "," ","Simplify the installation by compiling protobuf and openblas together with SINGA" + " "," ","Build python wheel automatically using Jenkins" + " "," ","Install SINGA from Debian packages" + "v1.2 April 2017 ","Numpy API ","Implement functions for the tensor module of PySINGA following numpy API" + " ","Distributed training ","Migrate distributed training frameworks from V0.3" + "v1.3 July 2017 ","Memory optimization ","Replace CNMEM with new memory pool to reduce memory footprint" + " ","Execution optimization ","Runtime optimization of execution scheduling" http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/rat-excludes ---------------------------------------------------------------------- diff --git a/rat-excludes b/rat-excludes index f7f79d2..9dd92e6 100644 --- a/rat-excludes +++ b/rat-excludes @@ -6,9 +6,8 @@ configure doc/* config/* \.dirstamp -autom4te.cache/* config.* -libtool stamp-h1 .*\.conf .*\.md +control http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/thirdparty/install.sh ---------------------------------------------------------------------- diff --git a/thirdparty/install.sh b/thirdparty/install.sh deleted file mode 100755 index bc3c159..0000000 --- a/thirdparty/install.sh +++ /dev/null @@ -1,714 +0,0 @@ -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ -#!/usr/bin/env bash - -function install_cmake() -{ - if [ ! -e "cmake-3.2.1.tar.gz" ] - then - wget http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/thirdparty/cmake-3.2.1.tar.gz; - fi - - rm -rf cmake-3.2.1; - tar zxvf cmake-3.2.1.tar.gz && cd cmake-3.2.1; - - if [ $# == 1 ] - then - echo "install cmake in $1"; - ./configure --prefix=$1; - make && make install; - elif [ $# == 0 ] - then - echo "install cmake in default path"; - ./configure; - make && sudo make install; - else - echo "wrong commands"; - fi - - if [ $? -ne 0 ] - then - cd ..; - return -1; - fi - - PATH=$1/bin:$PATH; - echo $PATH; - export PATH=$1/bin:$PATH; - cd ..; - return 0; -} - -function install_czmq() -{ - if [ ! -e "czmq-3.0.0-rc1.tar.gz" ] - then - wget http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/thirdparty/czmq-3.0.0-rc1.tar.gz; - fi - rm -rf czmq-3.0.0; - tar zxvf czmq-3.0.0-rc1.tar.gz && cd czmq-3.0.0; - - if [ $# == 2 ] - then - if [ $1 == "null" ] - then - echo "install czmq in default path. libzmq path is $2"; - ./configure --with-libzmq=$2; - make && sudo make install; - else - echo "install czmq in $1. libzmq path is $2"; - ./configure --prefix=$1 --with-libzmq=$2; - make && make install; - fi - elif [ $# == 1 ] - then - if [ $1 == "null" ] - then - echo "install czmq in default path."; - ./configure; - make && sudo make install; - else - echo "install czmq in $1"; - ./configure --prefix=$1; - make && make install; - fi - else - echo "ERROR: wrong command."; - return -1; - fi - - if [ $? -ne 0 ] - then - cd ..; - return -1; - fi - cd ..; - return 0; -} - -function install_glog() -{ - if [ ! -e "glog-0.3.3.tar.gz" ] - then - wget http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/thirdparty/glog-0.3.3.tar.gz; - fi - - rm -rf glog-0.3.3; - tar zxvf glog-0.3.3.tar.gz && cd glog-0.3.3; - - if [ $# == 1 ] - then - echo "install glog in $1"; - ./configure --prefix=$1; - make && make install; - elif [ $# == 0 ] - then - echo "install glog in default path"; - ./configure; - make && sudo make install; - else - echo "wrong commands"; - fi - - if [ $? -ne 0 ] - then - cd ..; - return -1; - fi - cd ..; - return 0; -} - -function install_lmdb() -{ - if [ ! -e "lmdb-0.9.10.tar.gz" ] - then - wget http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/thirdparty/lmdb-0.9.10.tar.gz; - fi - - rm -rf mdb-mdb; - tar zxvf lmdb-0.9.10.tar.gz && cd mdb-mdb/libraries/liblmdb; - - if [ $# == 1 ] - then - echo "install lmdb in $1"; - sed -i "26s#^.*#prefix=$1#g" Makefile; - make && make install; - elif [ $# == 0 ] - then - echo "install lmdb in default path"; - make && sudo make install; - else - echo "wrong commands"; - fi - - if [ $? -ne 0 ] - then - cd ../../..; - return -1; - fi - cd ../../..; - return 0; -} - -function install_openblas() -{ - if [ ! -e "OpenBLAS.zip" ] - then - wget http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/thirdparty/OpenBLAS.zip; - fi - - rm -rf OpenBLAS-develop; - unzip OpenBLAS.zip && cd OpenBLAS-develop; - - make ONLY_CBLAS=1; - if [ $? -ne 0 ] - then - cd ..; - return -1; - fi - if [ $# == 1 ] - then - echo "install OpenBLAS in $1"; - make PREFIX=$1 install; - if [ $? -ne 0 ] - then - cd ..; - return -1; - fi - elif [ $# == 0 ] - then - echo "install OpenBLAS in default path" - sudo make install; - if [ $? -ne 0 ] - then - cd ..; - return -1; - fi - else - echo "wrong commands"; - fi - cd ..; - - return 0; -} - -function install_opencv() -{ - if [ ! -e "opencv-2.4.10.zip" ] - then - wget http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/thirdparty/opencv-2.4.10.zip; - fi - - rm -rf opencv-2.4.10; - unzip opencv-2.4.10.zip && cd opencv-2.4.10; - - if [ $# == 1 ] - then - echo "install opencv in $1"; - cmake -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=$1; - make && make install; - elif [ $# == 0 ] - then - echo "install opencv in default path"; - cmake -DCMAKE_BUILD_TYPE=RELEASE; - make && sudo make install; - else - echo "wrong commands"; - fi - - if [ $? -ne 0 ] - then - cd ..; - return -1; - fi - cd ..; - return 0; -} - -function install_protobuf() -{ - if [ ! -e "protobuf-2.6.0.tar.gz" ] - then - wget http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/thirdparty/protobuf-2.6.0.tar.gz; - fi - - rm -rf protobuf-2.6.0; - tar zxvf protobuf-2.6.0.tar.gz && cd protobuf-2.6.0; - - if [ $# == 1 ] - then - echo "install protobuf in $1"; - ./configure --prefix=$1; - make && make install; - #cd python; - #python setup.py build; - #python setup.py install --prefix=$1; - #cd ..; - elif [ $# == 0 ] - then - echo "install protobuf in default path"; - ./configure; - make && sudo make install; - #cd python; - #python setup.py build; - #sudo python setup.py install; - #cd ..; - else - echo "wrong commands"; - fi - - if [ $? -ne 0 ] - then - cd ..; - return -1; - fi - cd ..; - return 0; -} - -function install_zeromq() -{ - if [ ! -e "zeromq-3.2.2.tar.gz" ] - then - wget http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/thirdparty/zeromq-3.2.2.tar.gz; - fi - - rm -rf zeromq-3.2.2; - tar zxvf zeromq-3.2.2.tar.gz && cd zeromq-3.2.2; - - if [ $# == 1 ] - then - echo "install zeromq in $1"; - ./configure --prefix=$1; - make && make install; - elif [ $# == 0 ] - then - echo "install zeromq in default path"; - ./configure; - make && sudo make install; - else - echo "wrong commands"; - fi - - if [ $? -ne 0 ] - then - cd ..; - return -1; - fi - cd ..; - return 0; -} - -function install_zookeeper() -{ - if [ ! -e "zookeeper-3.4.6.tar.gz" ] - then - wget http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/thirdparty/zookeeper-3.4.6.tar.gz; - fi - - rm -rf zookeeper-3.4.6; - tar zxvf zookeeper-3.4.6.tar.gz; - cd zookeeper-3.4.6/src/c; - - if [ $# == 1 ] - then - echo "install zookeeper in $1"; - ./configure --prefix=$1; - make && make install; - elif [ $# == 0 ] - then - echo "install zookeeper in default path"; - ./configure; - make && sudo make install; - else - echo "wrong commands"; - fi - - if [ $? -ne 0 ] - then - cd ../../..; - return -1; - fi - - cd ../../..; - return 0; -} - -BIN=`dirname "${BASH_SOURCE-$0}"` -BIN=`cd "$BIN">/dev/null; pwd` -BASE=`cd "$BIN/..">/dev/null; pwd` - -cd $BIN - -while [ $# != 0 ] -do - case $1 in -# "cmake") -# echo "install cmake"; -# if [[ $2 == */* ]];then -# install_cmake $2; -# if [ $? -ne 0 ] -# then -# echo "ERROR during cmake installation" ; -# exit; -# fi -# shift -# shift -# else -# install_cmake; -# if [ $? -ne 0 ] -# then -# echo "ERROR during cmake installation" ; -# exit; -# fi -# shift -# fi -# ;; - "czmq") - echo "install czmq"; - if [ $2 == "-f" ] - then - if [[ $4 == */* ]] - then - install_czmq $4 $3; - else - install_czmq null $3; - fi - if [ $? -ne 0 ] - then - echo "ERROR during czmq installation" ; - exit; - fi - shift - shift - shift - shift - elif [ $3 == "-f" ] - then - install_czmq $2 $4; - if [ $? -ne 0 ] - then - echo "ERROR during czmq installation" ; - exit; - fi - shift - shift - shift - shift - elif [[ $2 == */* ]] - then - install_czmq $2; - if [ $? -ne 0 ] - then - echo "ERROR during czmq installation" ; - exit; - fi - shift - shift - else - install_czmq null; - if [ $? -ne 0 ] - then - echo "ERROR during czmq installation" ; - exit; - fi - shift - fi - ;; - "glog") - echo "install glog"; - if [[ $2 == */* ]];then - install_glog $2; - if [ $? -ne 0 ] - then - echo "ERROR during glog installation" ; - exit; - fi - shift - shift - else - install_glog; - if [ $? -ne 0 ] - then - echo "ERROR during glog installation" ; - exit; - fi - shift - fi - ;; - "lmdb") - echo "install lmdb"; - if [[ $2 == */* ]];then - install_lmdb $2; - if [ $? -ne 0 ] - then - echo "ERROR during lmdb installation" ; - exit; - fi - shift - shift - else - install_lmdb; - if [ $? -ne 0 ] - then - echo "ERROR during lmdb installation" ; - exit; - fi - shift - fi - ;; - "OpenBLAS") - echo "install OpenBLAS"; - if [[ $2 == */* ]];then - install_openblas $2; - if [ $? -ne 0 ] - then - echo "ERROR during openblas installation" ; - exit; - fi - shift - shift - else - install_openblas; - if [ $? -ne 0 ] - then - echo "ERROR during openblas installation" ; - exit; - fi - shift - fi - ;; -# "opencv") -# echo "install opencv"; -# if [[ $2 == */* ]];then -# install_opencv $2; -# if [ $? -ne 0 ] -# then -# echo "ERROR during opencv installation" ; -# exit; -# fi -# shift -# shift -# else -# install_opencv; -# if [ $? -ne 0 ] -# then -# echo "ERROR during opencv installation" ; -# exit; -# fi -# shift -# fi -# ;; - "protobuf") - echo "install protobuf"; - if [[ $2 == */* ]];then - install_protobuf $2; - if [ $? -ne 0 ] - then - echo "ERROR during protobuf installation" ; - exit; - fi - shift - shift - else - install_protobuf; - if [ $? -ne 0 ] - then - echo "ERROR during protobuf installation" ; - exit; - fi - shift - fi - ;; - "zeromq") - echo "install zeromq"; - if [[ $2 == */* ]];then - install_zeromq $2; - if [ $? -ne 0 ] - then - echo "ERROR during zeromq installation" ; - exit; - fi - shift - shift - else - install_zeromq; - if [ $? -ne 0 ] - then - echo "ERROR during zeromq installation" ; - exit; - fi - shift - fi - ;; - "zookeeper") - echo "install zookeeper"; - if [[ $2 == */* ]];then - install_zookeeper $2; - if [ $? -ne 0 ] - then - echo "ERROR during zookeeper installation" ; - exit; - fi - shift - shift - else - install_zookeeper; - if [ $? -ne 0 ] - then - echo "ERROR during zookeeper installation" ; - exit; - fi - shift - fi - ;; - "all") - echo "install all dependencies"; - if [[ $2 == */* ]];then -# install_cmake $2; -# if [ $? -ne 0 ] -# then -# echo "ERROR during cmake installation" ; -# exit; -# fi - install_zeromq $2; - if [ $? -ne 0 ] - then - echo "ERROR during zeromq installation" ; - exit; - fi - install_czmq $2 $2; - if [ $? -ne 0 ] - then - echo "ERROR during czmq installation" ; - exit; - fi - install_glog $2; - if [ $? -ne 0 ] - then - echo "ERROR during glog installation" ; - exit; - fi -# install_lmdb $2; -# if [ $? -ne 0 ] -# then -# echo "ERROR during lmdb installation" ; -# exit; -# fi - install_openblas $2; - if [ $? -ne 0 ] - then - echo "ERROR during openblas installation" ; - exit; - fi -# install_opencv $2; -# if [ $? -ne 0 ] -# then -# echo "ERROR during opencv installation" ; -# exit; -# fi - install_protobuf $2; - if [ $? -ne 0 ] - then - echo "ERROR during protobuf installation" ; - exit; - fi - install_zookeeper $2; - if [ $? -ne 0 ] - then - echo "ERROR during zookeeper installation" ; - exit; - fi - shift - shift - else -# install_cmake; -# if [ $? -ne 0 ] -# then -# echo "ERROR during cmake installation" ; -# exit; -# fi - install_zeromq; - if [ $? -ne 0 ] - then - echo "ERROR during zeromq installation" ; - exit; - fi - install_czmq null; - if [ $? -ne 0 ] - then - echo "ERROR during czmq installation" ; - exit; - fi - install_glog; - if [ $? -ne 0 ] - then - echo "ERROR during glog installation" ; - exit; - fi -# install_lmdb; -# if [ $? -ne 0 ] -# then -# echo "ERROR during lmdb installation" ; -# exit; -# fi - install_openblas; - if [ $? -ne 0 ] - then - echo "ERROR during openblas installation" ; - exit; - fi -# install_opencv; -# if [ $? -ne 0 ] -# then -# echo "ERROR during opencv installation" ; -# exit; -# fi - install_protobuf; - if [ $? -ne 0 ] - then - echo "ERROR during protobuf installation" ; - exit; - fi - install_zookeeper; - if [ $? -ne 0 ] - then - echo "ERROR during zookeeper installation" ; - exit; - fi - shift - fi - ;; - *) - echo "USAGE: ./install.sh [MISSING_LIBRARY_NAME1] [YOUR_INSTALL_PATH1] [MISSING_LIBRARY_NAME2] [YOUR_INSTALL_PATH2] ..."; - echo " MISSING_LIBRARY_NAME can be: " -# echo " cmake" - echo " czmq" - echo " glog" - echo " lmdb" - echo " OpenBLAS" -# echo " opencv" - echo " protobuf" - echo " zeromq" - echo " zookeeper" - echo " To install all dependencies, you can run: " - echo " ./install.sh all" - exit; - esac -done http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/tool/graph.py ---------------------------------------------------------------------- diff --git a/tool/graph.py b/tool/graph.py deleted file mode 100644 index e2870ed..0000000 --- a/tool/graph.py +++ /dev/null @@ -1,40 +0,0 @@ -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ - -import sys -import pygraphviz -import networkx as nx -from networkx.readwrite import json_graph -import json - - -if __name__=='__main__': - print sys.argv - if len(sys.argv)<3: - print 'usage: draw the network graph\npython graph.py JSON_DAT FIG_FILE' - sys.exit() - - with open(sys.argv[1]) as fd: - nodelink=json.load(fd) - G=json_graph.node_link_graph(nodelink) - A = nx.to_agraph(G) - A.layout('dot', args='-Nfontsize=10 -Nwidth=".2" -Nheight=".2" -Nmargin=0 \ - -Gfontsize=8') - A.draw(sys.argv[2]) - - http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/tool/mesos/Makefile ---------------------------------------------------------------------- diff --git a/tool/mesos/Makefile b/tool/mesos/Makefile deleted file mode 100644 index d64fb5b..0000000 --- a/tool/mesos/Makefile +++ /dev/null @@ -1,25 +0,0 @@ -CXX_FLAGS=-I ../include -std=c++11 -I /usr/local/include/hdfs -I ../../include -LD_FLAGS=-lmesos -lsinga -lhdfs3 -EXE=scheduler -OBJS=singa_scheduler.o scheduler.pb.o -PROTOS=scheduler.proto -PROTO_HDRS=scheduler.pb.h -PROTO_SRCS=scheduler.pb.cc - -CXX=g++ - -all: $(PROTO_HDRS) $(EXE) - -$(PROTO_SRCS) $(PROTO_HDRS): $(PROTOS) - protoc --cpp_out=. $(PROTOS) - -$(EXE): $(OBJS) - $(CXX) -o $@ $(OBJS) $(LD_FLAGS) - rm -rf *.o - -%.o: %.cc - $(CXX) $(CXX_FLAGS) -c $< - -clean: - rm -rf *.o $(EXE) $(PROTO_HDRS) $(PROTO_SRCS) - http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/tool/mesos/README.md ---------------------------------------------------------------------- diff --git a/tool/mesos/README.md b/tool/mesos/README.md deleted file mode 100644 index 5330801..0000000 --- a/tool/mesos/README.md +++ /dev/null @@ -1,86 +0,0 @@ -This guide explains how to start SINGA distributed training on a Mesos cluster. It assumes that both Mesos and HDFS are -already running, and every node has SINGA installed. - -For this guide, we assume the following architecture, in which a cluster is set up from a set of containers (when the -cluster is already set up, the user can immediate [start the training](#job_start)). Refer to [Docker guide](https://github.com/ug93tad/incubator-singa/tree/SINGA-89/tool/docker/README.md) for details of how to start individual nodes and set up network connection between them (make sure [weave](http://weave.works/guides/weave-docker-ubuntu-simple.html) is running at each node, and the cluster's headnode is running in container `node0`) - -<img src="http://www.comp.nus.edu.sg/~dinhtta/files/singa_mesos.png"; align="center"> - -## Start HDFS and Mesos -Go inside each container, using: -```` -docker exec -it nodeX /bin/bash -```` -and configure it as follows: - -* On container `node0` - - hadoop namenode -format - hadoop-daemon.sh start namenode - /opt/mesos-0.22.0/build/bin/mesos-master.sh --work_dir=/opt --log_dir=/opt --quiet > /dev/null & - zk-service.sh start - -* On container `node1, node2, ...` - - hadoop-daemon.sh start datanode - /opt/mesos-0.22.0/build/bin/mesos-slave.sh --master=node0:5050 --log_dir=/opt --quiet > /dev/null & - -To check if the setup has been successful, check that HDFS namenode has registered `N` datanodes, via: - -```` -hadoop dfsadmin -report -```` - -#### Mesos logs -Mesos logs are stored at `/opt/lt-mesos-master.INFO` on `node0` and `/opt/lt-mesos-slave.INFO` at other nodes. - ---- - -## Starting SINGA training on Mesos -Assumed that Mesos and HDFS are already started, SINGA job can be launched at **any** container. - -#### Launching job - -1. Log in to any container, then - - cd incubator-singa/tool/mesos - -<a name="job_start"></a> -2. Check that configuration files are correct: - + `scheduler.conf` contains information about the master nodes - + `singa.conf` contains information about Zookeeper node - + Job configuration file `job.conf` **contains full path to the examples directories (NO RELATIVE PATH!).** - -3. Start the job: - + If starting for the first time: - - ./scheduler <job config file> -scheduler_conf <scheduler config file> -singa_conf <SINGA config file> - - + If not the first time: - - ./scheduler <job config file> - -**Notes.** Each running job is given a `frameworkID`. Look for the log message of the form: - - Framework registered with XXX-XXX-XXX-XXX-XXX-XXX - -#### Monitoring and Debugging - -Each Mesos job is given a `frameworkID` and a *sandbox* directory is created for each job. -The directory is in the specified `work_dir` (or `/tmp/mesos`) by default. For example, the error -during SINGA execution can be found at: - - /tmp/mesos/slaves/xxxxx-Sx/frameworks/xxxxx/executors/SINGA_x/runs/latest/stderr - -Other artifacts, like files downloaded from HDFS (`job.conf`) and `stdout` can be found in the same -directory. - -#### Stopping - -There are two way to kill the running job: - -1. If the scheduler is running in the foreground, simply kill it (using `Ctrl-C`, for example). -2. If the scheduler is running in the background, kill it using Mesos's REST API: - - curl -d "frameworkId=XXX-XXX-XXX-XXX-XXX-XXX" -X POST http://<master>/master/shutdown - http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/tool/mesos/docker/README.md ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/README.md b/tool/mesos/docker/README.md deleted file mode 100644 index fd33e7e..0000000 --- a/tool/mesos/docker/README.md +++ /dev/null @@ -1,189 +0,0 @@ -This guide explains how to set up a development environment for SINGA. A cluster environment can be launched over one or multiple physical nodes. - -1. [Build SINGA base](#build_base) -2. [Build SINGA with Mesos and Hadoop](#build_mesos) -3. [Pre-built images](#pre_built) -4. [Launch and stop SINGA (stand alone mode)](#launch_stand_alone) -5. [Launch pseudo-distributed SINGA on one node](#launch_pseudo) -6. [Launch fully distributed SINGA on multiple nodes](#launch_distributed) - ---- - -<a name="build_base"></a> -#### Build SINGA base image - -```` -$ cd tool/docker/singa -$ sudo docker build -t singa/base . -$ sudo docker images -REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE -singa/base latest XXXX XXX 2.01 GB -```` - -The result is the image containing a built version of SINGA. - -  - - *Figure 1. singa/base Docker image, containing library dependencies and SINGA built from source.* - --- - -<a name="build_mesos"></a> -#### Build SINGA with Mesos and Hadoop -```` -$ cd tool/docker/mesos -$ sudo docker build -t singa/mesos . -$ sudo docker images -REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE -singa/mesos latest XXXX XXX 4.935 GB -```` -  - - *Figure 2. singa/mesos Docker image, containing Hadoop and Mesos built on -top of SINGA. The default namenode address for Hadoop is `node0:9000`* - -**Notes** A common failure observed during the build process is caused by network failure occuring when downloading dependencies. Simply re-run the build command. - --- - -<a name="pre_built"></a> -#### Pre-built images on epiC cluster -For users with access to the `epiC` cluster, there are pre-built and loaded Docker images at the following nodes: - - ciidaa-c18 - ciidaa-c19 - -The available images at those nodes are: -```` -REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE -singa/base latest XXXX XXX 2.01 GB -singa/mesos latest XXXX XXX 4.935 GB -weaveworks/weaveexec 1.1.1 XXXX 11 days ago 57.8 MB -weaveworks/weave 1.1.1 XXXX 11 days ago 17.56 MB -```` - --- - -<a name="launch_stand_alone"></a> -#### Launch and stop SINGA in stand-alone mode -To launch a test environment for a single-node SINGA training, simply start a container from `singa/base` image. The following starts a container called -`XYZ`, then launches a shell in the container: - -```` -$ sudo docker run -dt --name XYZ singa/base /usr/bin/supervisord -$ sudo docker exec -it XYZ /bin/bash -```` - -  - - *Figure 3. Launch SINGA in stand-alone mode: single node training* - -Inside the launched container, the SINGA source directory can be found at `/root/incubator-singa`. - -**Stopping the container** - -```` -$ sudo docker stop XYZ -$ sudo docker rm ZYZ -```` - --- - -<a name="launch_pseudo"></a> -#### Launch SINGA on pseudo-distributed mode (single node) -To simulate a distributed environment on a single node, one can repeat the -previous step multiple times, each time giving a different name to the -container. Network connections between these containers are already supported, -thus SINGA instances/nodes in these container can readily communicate with each -other. - -The previous approach requires the user to start SINGA instances individually -at each container. Although there's a bash script for that, we provide a better -way. In particular, multiple containers can be started from `singa/mesos` image -which already bundles Mesos and Hadoop with SINGA. Using Mesos makes it easy to -launch, stop and monitor the distributed execution from a single container. -Figure 4 shows `N+1` containers running concurrently at the local host. - -```` -# sudo docker run -dt --name node0 singa/mesos /usr/bin/supervisord -# sudo docker run -dt --name node1 singa/mesos /usr/bin/supervisord -... -```` -  - - *Figure 4. Launch SINGA in pseudo-distributed mode : multiple SINGA nodes over one single machine* - -**Starting SINGA distributed training** - -Refer to the [Mesos -guide](https://github.com/ug93tad/incubator-singa/blob/SINGA-11/tool/mesos/README.md) -for details of how to start training with multiple SINGA instances. - -**Important:** the container that assumes the role of Hadoop's namenode (and often Mesos's and Zookeeper's mater node as well) **must** be named `node0`. Otherwise, the user must log in to individual containers and change the Hadoop configuration separately. - --- - -<a name="launch_distributed"></a> -#### Launch SINGA on fully distributed mode (multiple nodes) -The previous section has explained how to start a distributed environment on a -single node. But running many containers on one node does not scale. When there -are multiple physical hosts available, it is better to distribute the -containers over them. - -The only extra requirement for the fully distributed mode, as compared with the -pseudo distributed mode, is that the containers from different hosts are able -to transparently communicate with each other. In the pseudo distributed mode, -the local docker engine takes care of such communication. Here, we rely on -[Weave](http://weave.works/guides/weave-docker-ubuntu-simple.html) to make the -communication transparent. The resulting architecture is shown below. - -  - - *Figure 5. Launch SINGA in fully distributed mode: multiple SINGA nodes over multiple machines* - -**Install Weave at all hosts** -``` -$ curl -L git.io/weave -o /usr/local/bin/weave -$ chmod a+x /usr/local/bin/weave -``` - -**Starting Weave** - -Suppose `node0` will be launched at host with IP `111.222.111.222`. - -* At host `111.222.111.222`: -```` -$ weave launch -$ eval "$(weave env)" //if there's error, do `sudo -s` and try again -```` -* At other hosts: -```` -$ weave launch 111.222.111.222 -$ eval "$(weave env)" //if there's error, do `sudo -s` and try again -```` - -**Starting containers** - -The user logs in to each host and starts the container (same as in [pseudo-distributed](#launch_pseudo) mode). Note that container acting as the head node of the cluster must be named `node0` (and be running at the host with IP `111.222.111.222`, for example). - -**_Important_:** when there are other containers sharing the same host as `node0`, say `node1` and `node2` for example, -there're additional changes to be made to `node1` and `node2`. Particularly, log in to each container and edit -`/etc/hosts` file: - -```` -# modified by weave -... -X.Y.Z node0 node0.bridge //<- REMOVE this line -.. -```` -This is to ensure that name resolutions (of `node0`'s address) from `node1` and `node2` are correct. By default, -containers of the same host resolves each other's addresses via the Docker bridge. Instead, we want they to use -addressed given by Weave. - - -**Starting SINGA distributed training** - -Refer to the [Mesos -guide](https://github.com/ug93tad/incubator-singa/blob/SINGA-11/tool/mesos/README.md) -for details of how to start training with multiple SINGA instances. - http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/tool/mesos/docker/mesos/.bashrc ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/mesos/.bashrc b/tool/mesos/docker/mesos/.bashrc deleted file mode 100644 index c796a27..0000000 --- a/tool/mesos/docker/mesos/.bashrc +++ /dev/null @@ -1,32 +0,0 @@ -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ - -export LIBRARY_PATH=/opt/OpenBLAS/lib:$LIBRARY_PATH -export LD_LIBRARY_PATH=/opt/OpenBLAS/lib:/usr/local/lib:$LD_LIBRARY_PATH -export CPLUS_INCLUDE_PATH=/opt/OpenBLAS/include:$CPLUS_INCLUDE_PATH -export PATH=/opt/jdk1.8.0_60/bin:/opt/bin:$PATH -export HADOOP_HOME=/opt/hadoop-2.6.0 -export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin -export JAVA_HOME=/opt/jdk1.8.0_60 -alias ls="ls --color=always" -# some more ls aliases -alias ll="ls -alF" -alias la="ls -A" -alias l="ls -CF" -export SINGA_HOME=/root/incubator-singa -export PATH=$PATH:$SINGA_HOME/bin http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/tool/mesos/docker/mesos/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/mesos/Dockerfile b/tool/mesos/docker/mesos/Dockerfile deleted file mode 100644 index 83142b3..0000000 --- a/tool/mesos/docker/mesos/Dockerfile +++ /dev/null @@ -1,40 +0,0 @@ -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ - -#build Mesos+Hadoop on top of SINGA -FROM singa/base -MAINTAINER Anh Dinh<[email protected]> - -COPY incubator-singa/tool/docker/mesos/.bashrc /root/.bashrc -COPY incubator-singa/tool/docker/mesos/install.sh /opt/install.sh -COPY incubator-singa/tool/mesos/* /root/incubator-singa/tool/mesos/ - -RUN cd /opt && source ./install.sh - -RUN apt-get -y install curl cmake libxml2 libxml2-dev uuid-dev protobuf-compiler libprotobuf-dev libgsasl7-dev libkrb5-dev libboost1.54-all-dev - -RUN cd /opt && source /root/.bashrc && wget -c http://www.eu.apache.org/dist/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz && tar -zxvf hadoop-2.6.0.tar.gz && git clone https://github.com/PivotalRD/libhdfs3.git && cd libhdfs3 && mkdir build && cd build && cmake ../ && make && make install - -#to be removed after SINGA-11 is merged (pulling from Anh's branch instead of pulling from the master) -#RUN source ~/.bashrc && mkdir /root/mesos && cd /root/mesos && git clone https://github.com/ug93tad/incubator-singa && cd incubator-singa && git checkout SINGA-11 && cp -r tool/mesos /root/incubator-singa/tool/ && cd /root/incubator-singa/tool/mesos && make - -# update SINGA, but keep the examples directory -RUN source ~/.bashrc && cd /root/incubator-singa/tool/mesos && make - -COPY incubator-singa/tool/docker/mesos/*.xml /opt/hadoop-2.6.0/etc/hadoop/ -RUN ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa && cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys && echo 'StrictHostKeyChecking no' >> ~/.ssh/config http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/tool/mesos/docker/mesos/core-site.xml ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/mesos/core-site.xml b/tool/mesos/docker/mesos/core-site.xml deleted file mode 100644 index 49a39ca..0000000 --- a/tool/mesos/docker/mesos/core-site.xml +++ /dev/null @@ -1,24 +0,0 @@ -<?xml version="1.0" encoding="UTF-8"?> -<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> -<!-- - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. See accompanying LICENSE file. ---> - -<!-- Put site-specific property overrides in this file. --> - -<configuration> - <property> - <name>fs.default.name</name> - <value>hdfs://node0:9000</value> - </property> -</configuration> http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/tool/mesos/docker/mesos/hdfs-site.xml ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/mesos/hdfs-site.xml b/tool/mesos/docker/mesos/hdfs-site.xml deleted file mode 100644 index 2e0f4c6..0000000 --- a/tool/mesos/docker/mesos/hdfs-site.xml +++ /dev/null @@ -1,34 +0,0 @@ -<?xml version="1.0" encoding="UTF-8"?> -<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> -<!-- - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. See accompanying LICENSE file. ---> - -<!-- Put site-specific property overrides in this file. --> - -<configuration> -<property> - <name>dfs.replication</name> - <value>1</value> - </property> - -<property> - <name>dfs.name.dir</name> - <value>file:///var/lib/hadoop/hdfs/namenode</value> - </property> - - <property> - <name>dfs.data.dir</name> - <value>file:///var/lib/hadoop/hdfs/datanode</value> - </property> -</configuration> http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/tool/mesos/docker/mesos/install.sh ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/mesos/install.sh b/tool/mesos/docker/mesos/install.sh deleted file mode 100644 index 4b6ae79..0000000 --- a/tool/mesos/docker/mesos/install.sh +++ /dev/null @@ -1,35 +0,0 @@ -#!/bin/bash - -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ - -source /root/.bashrc -#download -cd /opt -wget -c http://archive.apache.org/dist/mesos/0.22.0/mesos-0.22.0.tar.gz -wget https://www.comp.nus.edu.sg/~dinhtta/files/mesos_patch -tar -zxvf mesos-0.22.0.tar.gz - -#patch and install mesos -cd /opt/mesos-0.22.0 -patch -p5 < ../mesos_patch -mkdir build; cd build -../configure -make -sudo make install - http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/tool/mesos/docker/mesos/mapred-site.xml ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/mesos/mapred-site.xml b/tool/mesos/docker/mesos/mapred-site.xml deleted file mode 100644 index 42bb81c..0000000 --- a/tool/mesos/docker/mesos/mapred-site.xml +++ /dev/null @@ -1,22 +0,0 @@ -<?xml version="1.0"?> -<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> -<!-- - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. See accompanying LICENSE file. ---> - -<!-- Put site-specific property overrides in this file. --> - -<configuration> - <property> - <name>mapreduce.framework.name</name> - <value>yarn</value> - </property> -</configuration> http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/tool/mesos/docker/mesos/yarn-site.xml ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/mesos/yarn-site.xml b/tool/mesos/docker/mesos/yarn-site.xml deleted file mode 100644 index c137af9..0000000 --- a/tool/mesos/docker/mesos/yarn-site.xml +++ /dev/null @@ -1,32 +0,0 @@ -<?xml version="1.0"?> -<!-- - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. See accompanying LICENSE file. ---> -<configuration> - -<!-- Site specific YARN configuration properties --> - <property> - <name>yarn.nodemanager.aux-services</name> - <value>mapreduce_shuffle</value> - </property> - - <property> - <name>yarn.resourcemanager.hostname</name> - <value>node0</value> - </property> - - <property> - <name>yarn.resourcemanager.bind-host</name> - <value>0.0.0.0</value> - </property> -</configuration> http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/tool/mesos/docker/singa/.bashrc ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/singa/.bashrc b/tool/mesos/docker/singa/.bashrc deleted file mode 100644 index cf0712f..0000000 --- a/tool/mesos/docker/singa/.bashrc +++ /dev/null @@ -1,30 +0,0 @@ -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ - -export LIBRARY_PATH=/opt/OpenBLAS/lib:$LIBRARY_PATH -export LD_LIBRARY_PATH=/opt/OpenBLAS/lib:/usr/local/lib:$LD_LIBRARY_PATH -export CPLUS_INCLUDE_PATH=/opt/OpenBLAS/include:$CPLUS_INCLUDE_PATH -export PATH=$PATH:/opt/jdk1.8.0_60/bin:/opt/bin -export JAVA_HOME=/opt/jdk1.8.0_60 -export SINGA_HOME=/root/incubator-singa -export PATH=$PATH:$SINGA_HOME/bin -alias ls="ls --color=always" -# some more ls aliases -alias ll="ls -alF" -alias la="ls -A" -alias l="ls -CF" http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/tool/mesos/docker/singa/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/singa/Dockerfile b/tool/mesos/docker/singa/Dockerfile deleted file mode 100644 index 0fc16bc..0000000 --- a/tool/mesos/docker/singa/Dockerfile +++ /dev/null @@ -1,34 +0,0 @@ -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ - -#build singa from scratch -FROM ubuntu:14.04 -MAINTAINER Anh Dinh<[email protected]> -RUN rm /bin/sh && ln -s /bin/bash /bin/sh - -RUN apt-get update && apt-get -y install g++-4.8 build-essential git vim wget zip automake libtool man python-dev python-boto libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev openssh-server supervisor - -RUN cd /opt && wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u60-b27/jdk-8u60-linux-x64.tar.gz && cd /opt && tar -zxvf jdk-8u60-linux-x64.tar.gz && rm -rf jdk-8u60-linux-x64.tar.gz - -COPY incubator-singa/tool/docker/singa/ssh.conf /etc/supervisor/conf.d/ -COPY incubator-singa/tool/docker/singa/.bashrc /root/.bashrc -COPY incubator-singa/tool/docker/singa/.vimrc /root/.vimrc -ADD incubator-singa /root/incubator-singa - -RUN cd && source /root/.bashrc && cd incubator-singa && ./thirdparty/install.sh all && cd thirdparty/ && wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz && tar -zxvf protobuf-2.5.0.tar.gz && cd protobuf-2.5.0 && ./configure && make clean && make && make install && cd ../../ && cp thirdparty/install.sh . && rm -rf thirdparty/* && mv install.sh thirdparty/ && ./thirdparty/install.sh zookeeper && ./autogen.sh && ./configure && make && make install && cd examples/cifar10 && mv Makefile.example Makefile && make download && make create && service ssh start - http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/tool/mesos/docker/singa/Dockerfile_gpu ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/singa/Dockerfile_gpu b/tool/mesos/docker/singa/Dockerfile_gpu deleted file mode 100644 index da57479..0000000 --- a/tool/mesos/docker/singa/Dockerfile_gpu +++ /dev/null @@ -1,80 +0,0 @@ -#/** -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ - -FROM singa/base -MAINTAINER NVIDIA CORPORATION <[email protected]> - -LABEL com.nvidia.volumes.needed="nvidia_driver" - -ENV NVIDIA_GPGKEY_SUM bd841d59a27a406e513db7d405550894188a4c1cd96bf8aa4f82f1b39e0b5c1c -ENV NVIDIA_GPGKEY_FPR 889bee522da690103c4b085ed88c3d385c37d3be - -RUN apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/GPGKEY && \ - apt-key adv --export --no-emit-version -a $NVIDIA_GPGKEY_FPR | tail -n +2 > cudasign.pub && \ - echo "$NVIDIA_GPGKEY_SUM cudasign.pub" | sha256sum -c --strict - && rm cudasign.pub && \ - echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64 /" > /etc/apt/sources.list.d/cuda.list - -ENV CUDA_VERSION 7.0 -LABEL com.nvidia.cuda.version="7.0" - -ENV CUDA_PKG_VERSION 7-0=7.0-28 -RUN apt-get update && apt-get install -y --no-install-recommends --force-yes curl \ - cuda-nvrtc-$CUDA_PKG_VERSION \ - cuda-cusolver-$CUDA_PKG_VERSION \ - cuda-cublas-$CUDA_PKG_VERSION \ - cuda-cufft-$CUDA_PKG_VERSION \ - cuda-curand-$CUDA_PKG_VERSION \ - cuda-cusparse-$CUDA_PKG_VERSION \ - cuda-npp-$CUDA_PKG_VERSION \ - cuda-cudart-$CUDA_PKG_VERSION && \ - ln -s cuda-$CUDA_VERSION /usr/local/cuda - -RUN apt-get install -y --no-install-recommends --force-yes \ - cuda-core-$CUDA_PKG_VERSION \ - cuda-misc-headers-$CUDA_PKG_VERSION \ - cuda-command-line-tools-$CUDA_PKG_VERSION \ - cuda-license-$CUDA_PKG_VERSION \ - cuda-nvrtc-dev-$CUDA_PKG_VERSION \ - cuda-cusolver-dev-$CUDA_PKG_VERSION \ - cuda-cublas-dev-$CUDA_PKG_VERSION \ - cuda-cufft-dev-$CUDA_PKG_VERSION \ - cuda-curand-dev-$CUDA_PKG_VERSION \ - cuda-cusparse-dev-$CUDA_PKG_VERSION \ - cuda-npp-dev-$CUDA_PKG_VERSION \ - cuda-cudart-dev-$CUDA_PKG_VERSION \ - cuda-driver-dev-$CUDA_PKG_VERSION - -RUN echo "/usr/local/cuda/lib" >> /etc/ld.so.conf.d/cuda.conf && \ - echo "/usr/local/cuda/lib64" >> /etc/ld.so.conf.d/cuda.conf && \ - ldconfig - -ENV CUDNN_DOWNLOAD_SUM 98679d5ec039acfd4d81b8bfdc6a6352d6439e921523ff9909d364e706275c2b - -RUN curl -fsSL http://developer.download.nvidia.com/compute/redist/cudnn/v3/cudnn-7.0-linux-x64-v3.0-prod.tgz -O && \ - echo "$CUDNN_DOWNLOAD_SUM cudnn-7.0-linux-x64-v3.0-prod.tgz" | sha256sum -c --strict - && \ - tar -xzf cudnn-7.0-linux-x64-v3.0-prod.tgz -C /usr/local && \ - rm cudnn-7.0-linux-x64-v3.0-prod.tgz && \ - ldconfig - -RUN echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf && \ - echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf - -ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH} -ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64:${LD_LIBRARY_PATH} - -RUN cd && source /root/.bashrc && cd incubator-singa && ./configure --enable-cuda --with-cuda=/usr/local/cuda --enable-cudnn --with-cudnn=/usr/local/cuda && make && make install http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/tool/mesos/docker/singa/ssh.conf ---------------------------------------------------------------------- diff --git a/tool/mesos/docker/singa/ssh.conf b/tool/mesos/docker/singa/ssh.conf deleted file mode 100644 index ac04d7e..0000000 --- a/tool/mesos/docker/singa/ssh.conf +++ /dev/null @@ -1,5 +0,0 @@ -[supervisord] -nodaemon=true - -[program:ssh] -command=/usr/sbin/sshd -D http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/tool/mesos/scheduler.conf ---------------------------------------------------------------------- diff --git a/tool/mesos/scheduler.conf b/tool/mesos/scheduler.conf deleted file mode 100644 index 1ab5fba..0000000 --- a/tool/mesos/scheduler.conf +++ /dev/null @@ -1,3 +0,0 @@ -namenode: "node0:9000" -master: "node0:5050" -job_counter: 5 http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/b2e21c62/tool/mesos/scheduler.proto ---------------------------------------------------------------------- diff --git a/tool/mesos/scheduler.proto b/tool/mesos/scheduler.proto deleted file mode 100644 index 322f7bb..0000000 --- a/tool/mesos/scheduler.proto +++ /dev/null @@ -1,33 +0,0 @@ -/************************************************************ -* -* Licensed to the Apache Software Foundation (ASF) under one -* or more contributor license agreements. See the NOTICE file -* distributed with this work for additional information -* regarding copyright ownership. The ASF licenses this file -* to you under the Apache License, Version 2.0 (the -* "License"); you may not use this file except in compliance -* with the License. You may obtain a copy of the License at -* -* http://www.apache.org/licenses/LICENSE-2.0 -* -* Unless required by applicable law or agreed to in writing, -* software distributed under the License is distributed on an -* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -* KIND, either express or implied. See the License for the -* specific language governing permissions and limitations -* under the License. -* -*************************************************************/ - -/** - * Protocol Buffer message containing global information for the - * scheduler. - */ -message SchedulerProto{ - // HDFS namenode, for example node0:9000 - required string namenode=1; - // Mesos master node, for example node0:5050 - required string master=2; - // jobID, incremented at each run - required int32 job_counter=3; -}
![http://www.comp.nus.edu.sg/~dinhtta/files/singa_mesos.png"](http://www.comp.nus.edu.sg/~dinhtta/files/singa_mesos.png"){kind=link}
