SINGA-346 Update cudnn from V5 to V7 Separate the Dockerfile for conda building and native building.
Change the swig version to 3.0.8 which works for both python2.7 and 3.0 Move all Dockerfiles into tool/docker folder. Project: http://git-wip-us.apache.org/repos/asf/incubator-singa/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-singa/commit/98e9d2f6 Tree: http://git-wip-us.apache.org/repos/asf/incubator-singa/tree/98e9d2f6 Diff: http://git-wip-us.apache.org/repos/asf/incubator-singa/diff/98e9d2f6 Branch: refs/heads/master Commit: 98e9d2f61ab9a350cc8ef72ccc9f26377acd28de Parents: ef5fdaf Author: Wang Wei <[email protected]> Authored: Mon Apr 16 22:56:34 2018 +0800 Committer: Wang Wei <[email protected]> Committed: Wed Apr 18 22:37:56 2018 +0800 ---------------------------------------------------------------------- .travis.yml | 3 + cmake/Dependencies.cmake | 4 +- tool/conda/README.md | 16 ++-- tool/conda/build.sh | 11 ++- tool/docker/README.md | 2 + tool/docker/devel/Dockerfile | 36 -------- tool/docker/devel/conda/cuda/Dockerfile | 62 +++++++++++++ tool/docker/devel/cuda/Dockerfile | 39 -------- tool/docker/devel/native/centos6/Dockerfile | 51 ++++++++++ tool/docker/devel/native/ubuntu/Dockerfile | 36 ++++++++ tool/docker/devel/native/ubuntu/cuda/Dockerfile | 61 ++++++++++++ tool/docker/runtime/Dockerfile | 31 ++++++- tool/docker/runtime/cuda/Dockerfile | 31 ------- tool/jenkins/README.md | 97 ++++++++++---------- .../docker/devel/native/centos6/Dockerfile | 64 ------------- .../docker/devel/native/ubuntu/Dockerfile | 70 -------------- tool/jenkins/docker/runtime/Dockerfile | 51 ---------- tool/jenkins/gen_doc.sh | 37 ++++++++ tool/jenkins/jenkins_doc.sh | 37 -------- tool/jenkins/jenkins_test.sh | 57 ------------ tool/jenkins/test.sh | 59 ++++++++++++ 21 files changed, 403 insertions(+), 452 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/.travis.yml ---------------------------------------------------------------------- diff --git a/.travis.yml b/.travis.yml index eb9ff69..5a032df 100644 --- a/.travis.yml +++ b/.travis.yml @@ -19,6 +19,8 @@ sudo: required language: cpp + + matrix: include: - os: osx @@ -27,6 +29,7 @@ matrix: - os: linux dist: trusty compiler: gcc +# TODO(wangwei) include python27 and python36 # #addons: http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/cmake/Dependencies.cmake ---------------------------------------------------------------------- diff --git a/cmake/Dependencies.cmake b/cmake/Dependencies.cmake index d237468..0b7d8a0 100644 --- a/cmake/Dependencies.cmake +++ b/cmake/Dependencies.cmake @@ -128,11 +128,11 @@ IF(USE_PYTHON) set(Python_ADDITIONAL_VERSIONS 3.6) FIND_PACKAGE(PythonLibs 3 REQUIRED) FIND_PACKAGE(PythonInterp 3 REQUIRED) - FIND_PACKAGE(SWIG 3.0.10 REQUIRED) + FIND_PACKAGE(SWIG 3.0.10 REQUIRED) ELSE() FIND_PACKAGE(PythonLibs 2.7 REQUIRED) FIND_PACKAGE(PythonInterp 2.7 REQUIRED) - FIND_PACKAGE(SWIG 3.0.8 REQUIRED) + FIND_PACKAGE(SWIG 3.0.8 REQUIRED) ENDIF() ENDIF() http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/conda/README.md ---------------------------------------------------------------------- diff --git a/tool/conda/README.md b/tool/conda/README.md index d30cf84..f162cf9 100644 --- a/tool/conda/README.md +++ b/tool/conda/README.md @@ -1,11 +1,11 @@ -# Package SINGA using conda-build +# Package Singa using conda-build [conda-build](https://conda.io/docs/user-guide/tasks/build-packages/index.html) is a packaging tool like apt-get, which is associated with [anaconda cloud](https://anaconda.org/) for package management for both python and cpp libraries. ## Environment variables -Depending on the building evironment, export the corresponding build string. +Build string is a part of the conda package specification. We include the cuda and cudnn version in it if Singa is compiled with CUDA enabled. # for singa with gpu, e.g. cuda8.0-cudnn7.0.5 export BUILD_STR=cudax.y-cudnna.b.c @@ -14,16 +14,20 @@ Depending on the building evironment, export the corresponding build string. export BUILD_STR=cpu -To package SINGA with CUDA and CUDNN, +To package Singa with CUDA and CUDNN, export CUDNN_PATH=<path to cudnn folder> -this folder should include a subfolder `include/cudnn.h` for the header file, and another subfolder `lib64` for the shared libraries. The BUILD_STR and CUDNN_PATH must be consistent. For example, if CUDNN_PATH is set, then BUILD_STR must be like cudax.y-cudnna.b.c. +this folder should include a subfolder `include/cudnn.h` for the header file, and another subfolder `lib64` for the shared libraries. The BUILD_STR and CUDNN_PATH must be consistent. For example, if CUDNN_PATH is set, then BUILD_STR must be like cudax.y-cudnna.b.c. CUDNN must be provided if we want to compiled Singa with CUDA enabled. ## Instruction -After exporting the environment variables, execute the following command to compile SINGA and package it +After exporting the environment variables, execute the following command to compile Singa and package it conda-build . -You will see the package path from the screen output. \ No newline at end of file +You will see the package path from the screen output. + +To clean the cache + + conda clean -ay \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/conda/build.sh ---------------------------------------------------------------------- diff --git a/tool/conda/build.sh b/tool/conda/build.sh index 2027725..cb4592c 100644 --- a/tool/conda/build.sh +++ b/tool/conda/build.sh @@ -19,9 +19,9 @@ export CPLUS_INCLUDE_PATH=`python -c "from __future__ import print_function; import numpy; print(numpy.get_include())"`:$CPLUS_INCLUDE_PATH # to let cmake use the dependent libs installed by conda, including python -export CMAKE_PREFIX_PATH=$PREFIX -export CMAKE_INCLUDE_PATH=$PREFIX/include -export CMAKE_LIBRARY_PATH=$PREFIX/lib +export CMAKE_PREFIX_PATH=$PREFIX:$CMAKE_PREFIX_PATH +export CMAKE_INCLUDE_PATH=$PREFIX/include:$CMAKE_INCLUDE_PATH +export CMAKE_LIBRARY_PATH=$PREFIX/lib:$CMAKE_LIBRARY_PATH mkdir build cd build @@ -29,12 +29,13 @@ USE_CUDA=OFF if [ -z ${CUDNN_PATH+x} ]; then USE_CUDA=ON cp $CUDNN_PATH/include $PREFIX/include - cp -P $CUDNN_PATH/lib64/* $PREFIX/lib/ + cp -P $CUDNN_PATH/lib64/libcudnn.so* $PREFIX/lib/ fi PYTHON3=OFF # PY3K is set by conda -if [ "$PY3K" == "True" ]; then PYTHON3=ON; fi +if [ "$PY3K" == "1" ]; then PYTHON3=ON; fi +echo "PY3K = $PY3K" cmake -DCMAKE_INSTALL_PREFIX=$PREFIX -DUSE_CUDA=$USE_CUDA -DPYTHON3=$PYTHON3 .. make make install http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/docker/README.md ---------------------------------------------------------------------- diff --git a/tool/docker/README.md b/tool/docker/README.md index d766cbc..d61554f 100644 --- a/tool/docker/README.md +++ b/tool/docker/README.md @@ -2,6 +2,8 @@ ## Availabe images +TO BE UPDATED. + | Tag | OS version | devel/runtime | Device|CUDA/CUDNN| |:----|:-----------|:--------------|:------|:---------| |runtime| Ubuntu16.04|runtime|CPU|-| http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/docker/devel/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/docker/devel/Dockerfile b/tool/docker/devel/Dockerfile deleted file mode 100644 index c19738d..0000000 --- a/tool/docker/devel/Dockerfile +++ /dev/null @@ -1,36 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# Base unbuntu 16.04 image -FROM ubuntu:latest - -MAINTAINER incubator-singa [email protected] - -# install dependencies -RUN apt-get update \ - && apt-get install -y --no-install-recommends build-essential git cmake libprotobuf-dev libopenblas-dev protobuf-compiler python-dev python-pip swig wget\ - && apt-get clean && apt-get autoremove && apt-get autoclean \ - && rm -rf /var/lib/apt/lists/* \ - && pip --no-cache-dir install -U pip numpy setuptools - -# set environment -ENV CPLUS_INCLUDE_PATH /usr/local/lib/python2.7/dist-packages/numpy/core/include:${CPLUS_INCLUDE_PATH} - -# download singa source -RUN git clone https://github.com/apache/incubator-singa.git - -# compile singa and install pysinga -RUN cd incubator-singa && mkdir build && cd build && cmake .. && make && cd python && pip install . && cd ../../ && rm -rf build http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/docker/devel/conda/cuda/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/docker/devel/conda/cuda/Dockerfile b/tool/docker/devel/conda/cuda/Dockerfile new file mode 100644 index 0000000..ea360f3 --- /dev/null +++ b/tool/docker/devel/conda/cuda/Dockerfile @@ -0,0 +1,62 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Base unbuntu 14.04 image from nvidia/cuda +# Change tags to build with different cuda/cudnn versions: +FROM nvidia/cuda:9.0-devel-ubuntu16.04 +# FROM nvidia/cuda:7.5-cudnn5-devel-ubuntu14.04 +# FROM nvidia/cuda:7.5-cudnn4-devel-ubuntu14.04 +# FROM nvidia/cuda:7.0-cudnn4-devel-ubuntu14.04 +# FROM nvidia/cuda:7.5-cudnn5-devel-ubuntu14.04 + +# install dependencies +RUN apt-get update \ + && apt-get install -y --no-install-recommends git build-essential cmake wget openssh-server ca-certificates\ + && apt-get clean && apt-get autoremove && apt-get autoclean \ + && rm -rf /var/lib/apt/lists/* + +# install conda, conda-build and anaconda-client +RUN wget --no-check-certificate https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh; +RUN bash miniconda.sh -b -p /root/miniconda +RUN /root/miniconda/bin/conda config --set always_yes yes --set changeps1 no +RUN /root/miniconda/bin/conda update -q conda +RUN /root/miniconda/bin/conda install conda-build +RUN /root/miniconda/bin/conda install anaconda-client +ENV PATH /root/miniconda/bin:${PATH} + +# set environment +# ENV CMAKE_INCLUDE_PATH /usr/local/cuda/include:${CMAKE_INCLUDE_PATH} +# ENV CMAKE_LIBRARY_PATH /usr/local/cuda/lib64:${CMAKE_LIBRARY_PATH} + +# download singa source +# RUN git clone https://github.com/apache/incubator-singa.git + +# config ssh service +RUN mkdir /var/run/sshd +RUN echo 'root:singa' | chpasswd +# for ubuntu 14.04 +# RUN sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config +# for ubuntu 16.04 prohibit +RUN sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config +# SSH login fix. Otherwise user is kicked off after login +RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd + +# dump environment variables into files, so that ssh can see also +RUN env | grep _ >> /etc/environment + +EXPOSE 22 + +CMD ["/usr/sbin/sshd", "-D"] http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/docker/devel/cuda/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/docker/devel/cuda/Dockerfile b/tool/docker/devel/cuda/Dockerfile deleted file mode 100644 index d4b84b2..0000000 --- a/tool/docker/devel/cuda/Dockerfile +++ /dev/null @@ -1,39 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# Base unbuntu 16.04 image from nvidia/cuda -# Change tags to build with different cuda/cudnn versions: -# FROM nvidia/cuda:8.0-cudnn5-devel-ubuntu16.04 -FROM nvidia/cuda:8.0-cudnn5-devel-ubuntu16.04 - -MAINTAINER incubator-singa [email protected] - -# install dependencies -RUN apt-get update \ - && apt-get install -y --no-install-recommends git g++ cmake libprotobuf-dev libopenblas-dev protobuf-compiler python-dev python-pip swig wget\ - && apt-get clean && apt-get autoremove && apt-get autoclean \ - && rm -rf /var/lib/apt/lists/* \ - && pip --no-cache-dir install -U pip numpy setuptools - -# set environment -ENV CPLUS_INCLUDE_PATH /usr/local/lib/python2.7/dist-packages/numpy/core/include:${CPLUS_INCLUDE_PATH} -ENV CMAKE_INCLUDE_PATH /usr/local/cuda/include:${CMAKE_INCLUDE_PATH} -ENV CMAKE_LIBRARY_PATH /usr/local/cuda/lib64:${CMAKE_LIBRARY_PATH} - -# download singa source -RUN git clone https://github.com/apache/incubator-singa.git - -RUN cd incubator-singa && mkdir build && cd build && cmake -DUSE_CUDA=ON .. && make && cd python && pip install . && cd ../../ && rm -rf build http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/docker/devel/native/centos6/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/docker/devel/native/centos6/Dockerfile b/tool/docker/devel/native/centos6/Dockerfile new file mode 100644 index 0000000..0538c13 --- /dev/null +++ b/tool/docker/devel/native/centos6/Dockerfile @@ -0,0 +1,51 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Change tags to build with different cuda/cudnn versions: +FROM nvidia/cuda:7.5-cudnn5-devel-centos6 + +# install dependencies +RUN yum -y update && yum -y install git wget openssh-server cmake + + + +# set environment +ENV CPLUS_INCLUDE_PATH /usr/local/lib/python2.7/dist-packages/numpy/core/include:${CPLUS_INCLUDE_PATH} +# ENV CMAKE_INCLUDE_PATH /usr/local/cuda/include:${CMAKE_INCLUDE_PATH} +# ENV CMAKE_LIBRARY_PATH /usr/local/cuda/lib64:${CMAKE_LIBRARY_PATH} + +# download singa source +# RUN git clone https://github.com/apache/incubator-singa.git + +# config ssh service +RUN mkdir /var/run/sshd +RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key +RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key + +RUN sed -ri 's/UsePAM yes/#UsePAM yes/g' /etc/ssh/sshd_config +RUN sed -ri 's/#UsePAM no/UsePAM no/g' /etc/ssh/sshd_config +RUN echo 'root:singa' | chpasswd + +#RUN sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config +# SSH login fix. Otherwise user is kicked off after login +#RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd + +# dump environment variables into files, so that ssh can see also +RUN env | grep _ >> /etc/environment + +EXPOSE 22 + +CMD ["/usr/sbin/sshd", "-D"] http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/docker/devel/native/ubuntu/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/docker/devel/native/ubuntu/Dockerfile b/tool/docker/devel/native/ubuntu/Dockerfile new file mode 100644 index 0000000..c19738d --- /dev/null +++ b/tool/docker/devel/native/ubuntu/Dockerfile @@ -0,0 +1,36 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Base unbuntu 16.04 image +FROM ubuntu:latest + +MAINTAINER incubator-singa [email protected] + +# install dependencies +RUN apt-get update \ + && apt-get install -y --no-install-recommends build-essential git cmake libprotobuf-dev libopenblas-dev protobuf-compiler python-dev python-pip swig wget\ + && apt-get clean && apt-get autoremove && apt-get autoclean \ + && rm -rf /var/lib/apt/lists/* \ + && pip --no-cache-dir install -U pip numpy setuptools + +# set environment +ENV CPLUS_INCLUDE_PATH /usr/local/lib/python2.7/dist-packages/numpy/core/include:${CPLUS_INCLUDE_PATH} + +# download singa source +RUN git clone https://github.com/apache/incubator-singa.git + +# compile singa and install pysinga +RUN cd incubator-singa && mkdir build && cd build && cmake .. && make && cd python && pip install . && cd ../../ && rm -rf build http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/docker/devel/native/ubuntu/cuda/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/docker/devel/native/ubuntu/cuda/Dockerfile b/tool/docker/devel/native/ubuntu/cuda/Dockerfile new file mode 100644 index 0000000..03d53c0 --- /dev/null +++ b/tool/docker/devel/native/ubuntu/cuda/Dockerfile @@ -0,0 +1,61 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# +# Change tags to build with different cuda/cudnn versions: +FROM nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04 +# FROM nvidia/cuda:7.5-cudnn5-devel-ubuntu14.04 +# FROM nvidia/cuda:7.5-cudnn4-devel-ubuntu14.04 +# FROM nvidia/cuda:7.0-cudnn4-devel-ubuntu14.04 +# FROM nvidia/cuda:7.5-cudnn5-devel-ubuntu14.04 + +# install dependencies +RUN apt-get update \ + && apt-get install -y --no-install-recommends git build-essential autoconf libtool cmake libprotobuf-dev libopenblas-dev libpcre3-dev python-dev python-pip protobuf-compiler wget openssh-server swig\ + && apt-get clean && apt-get autoremove && apt-get autoclean \ + && rm -rf /var/lib/apt/lists/* \ + && pip install -U pip wheel numpy setuptools unittest-xml-reporting protobuf future + + +# install swig > 3.0.10 for ubuntu < 16.04 +# RUN wget http://prdownloads.sourceforge.net/swig/swig-3.0.10.tar.gz && \ +# tar zxf swig-3.0.10.tar.gz && cd swig-3.0.10 && \ +# ./configure && make && make install + +# set environment +ENV CPLUS_INCLUDE_PATH /usr/local/lib/python2.7/dist-packages/numpy/core/include:${CPLUS_INCLUDE_PATH} +# ENV CMAKE_INCLUDE_PATH /usr/local/cuda/include:${CMAKE_INCLUDE_PATH} +# ENV CMAKE_LIBRARY_PATH /usr/local/cuda/lib64:${CMAKE_LIBRARY_PATH} + +# download singa source +# RUN git clone https://github.com/apache/incubator-singa.git + +# config ssh service +RUN mkdir /var/run/sshd +RUN echo 'root:singa' | chpasswd +# for ubuntu 14.04 +# RUN sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config +# for ubuntu 16.04 prohibit +RUN sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config +# SSH login fix. Otherwise user is kicked off after login +RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd + +# dump environment variables into files, so that ssh can see also +RUN env | grep _ >> /etc/environment + +EXPOSE 22 + +CMD ["/usr/sbin/sshd", "-D"] http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/docker/runtime/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/docker/runtime/Dockerfile b/tool/docker/runtime/Dockerfile index 40c78b5..ff7d380 100644 --- a/tool/docker/runtime/Dockerfile +++ b/tool/docker/runtime/Dockerfile @@ -1,3 +1,5 @@ +#!/usr/bin/env sh +# # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information @@ -21,11 +23,30 @@ MAINTAINER incubator-singa [email protected] # install dependencies RUN apt-get update \ - && apt-get install -y --no-install-recommends git python python-pip \ + && apt-get install -y --no-install-recommends subversion git wget openssh-server bzip2\ && apt-get clean && apt-get autoremove && apt-get autoclean \ - && rm -rf /var/lib/apt/lists/* \ - && pip --no-cache-dir install -U pip setuptools + && rm -rf /var/lib/apt/lists/* + +RUN wget --no-check-certificate https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -O miniconda.sh; +RUN bash miniconda.sh -b -p /root/miniconda +ENV PATH /root/miniconda/bin:${PATH} +RUN conda config --set always_yes yes --set changeps1 no +RUN conda update -q conda +RUN conda install -c conda-forge sphinx +RUN conda install -c conda-forge sphinx_rtd_theme +RUN conda install -c clinicalgraphics recommonmark=0.2.0 +# TODO(wangwei) add the cpu build string +RUN conda install -c nusdbsystem singa + +RUN mkdir /var/run/sshd +RUN echo 'root:singa' | chpasswd +RUN sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config +# SSH login fix. Otherwise user is kicked off after login +RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd + +# dump environment variables into files, so that ssh can see also +# RUN env | grep _ >> /etc/environment +EXPOSE 22 -# install pysinga TODO(wangwei) install debian package -RUN pip install --upgrade http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/linux/latest/ubuntu16.04-cpp/singa-1.1.0-py2-none-any.whl +CMD ["/usr/sbin/sshd", "-D"] http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/docker/runtime/cuda/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/docker/runtime/cuda/Dockerfile b/tool/docker/runtime/cuda/Dockerfile deleted file mode 100644 index baf9bfd..0000000 --- a/tool/docker/runtime/cuda/Dockerfile +++ /dev/null @@ -1,31 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# Base unbuntu 16.04, cuda8.0, cudnn5 -FROM nvidia/cuda:8.0-cudnn5-runtime-ubuntu16.04 - -MAINTAINER incubator-singa [email protected] - -# install dependencies -RUN apt-get update \ - && apt-get install -y --no-install-recommends git python python-pip \ - && apt-get clean && apt-get autoremove && apt-get autoclean \ - && rm -rf /var/lib/apt/lists/* \ - && pip --no-cache-dir install -U pip setuptools - - -# install pysinga TODO(wangwei) install debian package -RUN pip install --upgrade http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/linux/latest/ubuntu16.04-cuda8.0-cudnn5/singa-1.1.0-py2-none-any.whl http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/jenkins/README.md ---------------------------------------------------------------------- diff --git a/tool/jenkins/README.md b/tool/jenkins/README.md index 6228c88..9114861 100644 --- a/tool/jenkins/README.md +++ b/tool/jenkins/README.md @@ -1,11 +1,12 @@ # Jenkins CI Support ## Introduction -This documentation is to guide SINGA developers to setup Jenkins service to support continuous integration on GPU systems. After each commit, -1. SINGA should be compiled and tested automatically under different settings (e.g., OS and hardware). -2. Convenient binaries should be generated automatically and archived. +This documentation is to guide Singa developers to setup Jenkins service for continuous integration of Singa. After each commit, +1. Singa should be compiled and tested automatically under different settings (e.g.,OS, python version and hardware). +2. Binary packages should be generated automatically and archived. Continuous integration for CPU systems is enabled via [Travis](../travis). +Hence, Jenkins is mainly used for CI on GPUs. ## Install Jenkins [Jenkins Official Wiki](https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins) @@ -21,18 +22,10 @@ Create a multi-configuration project and configure project as follows: ### Description This job automatically pulls latest commits from Apache incubator-singa github repository, then for different environments -* compile and test SINGA on GPUs -* create Debian GPU packages -* create anaconda GPU packages - -The working nodes (or Docker containers) are configured in Jenkins-Manage Jenkins-Mange Nodes. -Each node should configure the following environment variable -1. CUDA_VERSION, e.g., 7.5 -2. CUDNN_VERSION e.g, 5 -3. ANACONDA_UPLOAD_TOKEN -4. SINGA_NAME=singa-cuda${CUDA_VERSION}-cudnn${CUDNN_VERSION} -5. OS_VERSION, e.g., ubuntu14.04 -6. SINGA_INCLUDE_PATH and SINGA_LIBRARY_PATH for the cudnn.h and libcudnn.so folder respectively +* compile and test Singa on GPUs +* generate conda package of Singa with CUDA enabled +* invoke the CPU test and packaging on Travis +* (optional) create Debian GPU packages ### General * Discard old builds - Max # of builds to keep - 50 @@ -50,27 +43,22 @@ Each node should configure the following environment variable * Slave - name ``env`` Node/label: tick available nodes ### Build +The building script can do the following tasks: + * compile and do unit test on GPU - Execute shell - command - ``bash -ex tool/jenkins/test.sh $lang`` - `$lang` is set in **Configuration Matrix* section + Execute shell - command - ``bash -ex tool/jenkins/test.sh`` - * create Debian package - Execute shell - command - ``bash -ex tool/debian/build.sh --python --$lang`` + * update another github repo with the new commits to invoke travis (for cpu test and conda package generation) + Execute shell - command - ``git push https://<username:token>@github.com/nusdbsystem/incubator-singa.git -f`` - * create conda package - Execute shell - command - + * create conda package and upload it to anaconda cloud + Execute shell - command - git push https://username:[email protected]/nusdbsystem/incubator-singa.git -f - bash -ex tool/jenkins/jenkins_test.sh $lang - export CONDA_BLD_PATH=/root/conda-bld-$BUILD_NUMBER - mkdir $CONDA_BLD_PATH /root/miniconda/bin/conda-build tool/conda - /root/miniconda/bin/anaconda -t ANACONDA_UPLOAD_TOKEN upload -u nusdbsystem -l main $CONDA_BLD_PATH/linux-64/singa-*.tar.bz2 --force - + /root/miniconda/bin/anaconda -t <ANACONDA_UPLOAD_TOKEN> upload -u nusdbsystem -l main /root/miniconda/linux-64/singa-*.so.*.tar.bz2 --force - It first pushes to a mirror site to invoke travis-ci for CPU package creation; - Then it compiles and runs unit tests; - Finally it creates the conda package for GPU and upload it. + * (optional) create Debian package + Execute shell - command - ``bash -ex tool/debian/build.sh --python --$lang`` ### Post-build Actions * Publish JUnit test result report - Test report XMLs - ``**/gtest.xml, **/unittest.xml`` @@ -91,59 +79,70 @@ Each node should configure the following environment variable debian/32/84d56b7/ubuntu14.04-cpp/singa-1.0.1.deb debian/32/84d56b7/ubuntu14.04-cuda8.0-cudnn5/singa-1.0.1.deb -### Docker Images -We provide a number of singa docker [images](./docker) for Jenkins to use as slaves. +### Jenkins Nodes + +We provide different Singa [Dockerfiles](../docker/README.md) for Jenkins to use as working nodes. + To run the docker images, - nvidia-docker run --name <jenkins-slaveXX> -d <Image ID> -P + nvidia-docker run --name <node name> -d <Image ID> -P To add the container into a network for easy access docker network create <network name> - docker network connect <network name> <jenkins-slaveXX> + docker network connect <network name> <node name> -After connecting both the jenkins and slave contaniners into the same network, we can ssh to the slave from jenkins container like +After connecting both the jenkins and node contaniners into the same network, we can ssh to the node from jenkins container like # inside jenkins container - ssh root@<jenkins-slaveXX> + ssh root@<node name> -You need execute the above command manually for the first ssh login +You need execute the above command manually for the first ssh login. In the Jenkins node configuration page, the container name is used to configure the `Host` field. Notice that Oracle username and account are required to luanch the node by Jenkins. +The working nodes (or Docker containers) are configured in Jenkins-Manage Jenkins-Mange Nodes. +Each node should configure the following environment variable + + export CUDNN_PATH=<path to cudnn folder> -## Configure Jenkins for SINGA Website Updates +where the cudnn folder should include `inlcude/cudnn.h` and `lib64/libcudnn.so*`. [Dockerfiles](../docker/README.md) are provided to create the working nodes. + +## Configure Jenkins for Singa Website Updates ### Description and Configuration This job is triggered upon any changes to the files of the `doc/` folder. It does the following tasks, -1. installs the latest PySINGA +1. installs the latest Singa 2. pull the latest source code 3. generate the html files for the documentation -4. update the SINGA website +4. update the Singa website The Jenkins job configuration is similar as above except the following fields, * Source Code Management - Git - Additional Behaviors - Include Region `doc/*` -* Build - Execute Shell - Command `bash -ex tool/jenkins/jenkins_doc.sh` -* No `Post-build Actions` +* Build - Execute Shell - Command -### Docker Images + bash -ex tool/jenkins/gen_doc.sh -The Docker image for the Jenkins slave node is at `docker/ubuntu16.04/runtime/Dockerfile`. -To build the docker image, +* No `Post-build Actions` - # under the docker/ubuntu16.04/runtime/ folder - $ docker built -t singa:doc . +### Jenkins Node -To start the slave node +The docker images used for testing also be used for document generation. +We have to manually configure something inside the docker container. +First, we start the container - $ docker run --name singa-doc -d singa:doc + $docker run --name singa-doc -d <docker image> + # docker network connect jenkins singa-doc $ docker exec -it singa-doc /bin/bash + +Next, we do the first commit to the svn repo. + $ svn co https://svn.apache.org/repos/asf/incubator/singa/site/trunk # update ~/.subversion/config to set 'store-password=yes' # to set password free commit, we have to do a manual commit at first. http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/jenkins/docker/devel/native/centos6/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/jenkins/docker/devel/native/centos6/Dockerfile b/tool/jenkins/docker/devel/native/centos6/Dockerfile deleted file mode 100644 index 61b30f6..0000000 --- a/tool/jenkins/docker/devel/native/centos6/Dockerfile +++ /dev/null @@ -1,64 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# Base unbuntu 14.04 image from nvidia/cuda -# Change tags to build with different cuda/cudnn versions: -# FROM nvidia/cuda:8.0-cudnn5-devel-ubuntu14.04 -# FROM nvidia/cuda:7.5-cudnn5-devel-ubuntu14.04 -# FROM nvidia/cuda:7.5-cudnn4-devel-ubuntu14.04 -# FROM nvidia/cuda:7.0-cudnn4-devel-ubuntu14.04 -FROM nvidia/cuda:7.5-cudnn5-devel-centos6 - -# install dependencies -RUN yum -y update && yum -y install git wget openssh-server cmake - - -# install conda, conda-build and anaconda-client -RUN wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -O miniconda.sh; -RUN bash miniconda.sh -b -p /root/miniconda -ENV PATH /root/miniconda/bin:${PATH} -RUN /root/miniconda/bin/conda config --set always_yes yes --set changeps1 no -RUN /root/miniconda/bin/conda update -q conda -RUN /root/miniconda/bin/conda install conda-build -RUN /root/miniconda/bin/conda install anaconda-client - -# set environment -ENV CPLUS_INCLUDE_PATH /usr/local/lib/python2.7/dist-packages/numpy/core/include:${CPLUS_INCLUDE_PATH} -# ENV CMAKE_INCLUDE_PATH /usr/local/cuda/include:${CMAKE_INCLUDE_PATH} -# ENV CMAKE_LIBRARY_PATH /usr/local/cuda/lib64:${CMAKE_LIBRARY_PATH} - -# download singa source -# RUN git clone https://github.com/apache/incubator-singa.git - -# config ssh service -RUN mkdir /var/run/sshd -RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key - -RUN sed -ri 's/UsePAM yes/#UsePAM yes/g' /etc/ssh/sshd_config -RUN sed -ri 's/#UsePAM no/UsePAM no/g' /etc/ssh/sshd_config -RUN echo 'root:singa' | chpasswd - -#RUN sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config -# SSH login fix. Otherwise user is kicked off after login -#RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd - -# dump environment variables into files, so that ssh can see also -RUN env | grep _ >> /etc/environment - -EXPOSE 22 - -CMD ["/usr/sbin/sshd", "-D"] http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/jenkins/docker/devel/native/ubuntu/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/jenkins/docker/devel/native/ubuntu/Dockerfile b/tool/jenkins/docker/devel/native/ubuntu/Dockerfile deleted file mode 100644 index 302c579..0000000 --- a/tool/jenkins/docker/devel/native/ubuntu/Dockerfile +++ /dev/null @@ -1,70 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# Base unbuntu 14.04 image from nvidia/cuda -# Change tags to build with different cuda/cudnn versions: -FROM nvidia/cuda:8.0-cudnn5-devel-ubuntu16.04 -# FROM nvidia/cuda:7.5-cudnn5-devel-ubuntu14.04 -# FROM nvidia/cuda:7.5-cudnn4-devel-ubuntu14.04 -# FROM nvidia/cuda:7.0-cudnn4-devel-ubuntu14.04 -# FROM nvidia/cuda:7.5-cudnn5-devel-ubuntu14.04 - -# install dependencies -RUN apt-get update \ - && apt-get install -y --no-install-recommends git build-essential autoconf libtool cmake libprotobuf-dev libopenblas-dev libpcre3-dev python-dev python-pip protobuf-compiler wget openssh-server swig\ - && apt-get clean && apt-get autoremove && apt-get autoclean \ - && rm -rf /var/lib/apt/lists/* \ - && pip install -U pip wheel numpy setuptools unittest-xml-reporting protobuf future - - -# install swig > 3.0.10 for ubuntu < 16.04 -# RUN wget http://prdownloads.sourceforge.net/swig/swig-3.0.10.tar.gz && \ -# tar zxf swig-3.0.10.tar.gz && cd swig-3.0.10 && \ -# ./configure && make && make install - -# install conda, conda-build and anaconda-client -RUN wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -O miniconda.sh; -RUN bash miniconda.sh -b -p /root/miniconda -RUN /root/miniconda/bin/conda config --set always_yes yes --set changeps1 no -RUN /root/miniconda/bin/conda update -q conda -RUN /root/miniconda/bin/conda install conda-build -RUN /root/miniconda/bin/conda install anaconda-client -ENV PATH /root/miniconda/bin:${PATH} - -# set environment -ENV CPLUS_INCLUDE_PATH /usr/local/lib/python2.7/dist-packages/numpy/core/include:${CPLUS_INCLUDE_PATH} -# ENV CMAKE_INCLUDE_PATH /usr/local/cuda/include:${CMAKE_INCLUDE_PATH} -# ENV CMAKE_LIBRARY_PATH /usr/local/cuda/lib64:${CMAKE_LIBRARY_PATH} - -# download singa source -# RUN git clone https://github.com/apache/incubator-singa.git - -# config ssh service -RUN mkdir /var/run/sshd -RUN echo 'root:singa' | chpasswd -# for ubuntu 14.04 -# RUN sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config -# for ubuntu 16.04 prohibit -RUN sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config -# SSH login fix. Otherwise user is kicked off after login -RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd - -# dump environment variables into files, so that ssh can see also -RUN env | grep _ >> /etc/environment - -EXPOSE 22 - -CMD ["/usr/sbin/sshd", "-D"] http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/jenkins/docker/runtime/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/jenkins/docker/runtime/Dockerfile b/tool/jenkins/docker/runtime/Dockerfile deleted file mode 100644 index 9f4aea3..0000000 --- a/tool/jenkins/docker/runtime/Dockerfile +++ /dev/null @@ -1,51 +0,0 @@ -#!/usr/bin/env sh -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# Base unbuntu 16.04 image -FROM ubuntu:latest - -MAINTAINER incubator-singa [email protected] - -# install dependencies -RUN apt-get update \ - && apt-get install -y --no-install-recommends subversion git wget openssh-server bzip2\ - && apt-get clean && apt-get autoremove && apt-get autoclean \ - && rm -rf /var/lib/apt/lists/* - -RUN wget --no-check-certificate https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -O miniconda.sh; -RUN bash miniconda.sh -b -p /root/miniconda -ENV PATH /root/miniconda/bin:${PATH} -RUN conda config --set always_yes yes --set changeps1 no -RUN conda update -q conda -RUN conda install -c conda-forge sphinx -RUN conda install -c conda-forge sphinx_rtd_theme -RUN conda install -c clinicalgraphics recommonmark=0.2.0 -RUN conda install -c nusdbsystem singa - -RUN mkdir /var/run/sshd -RUN echo 'root:singa' | chpasswd -RUN sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config -# SSH login fix. Otherwise user is kicked off after login -RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd - -# dump environment variables into files, so that ssh can see also -# RUN env | grep _ >> /etc/environment - -EXPOSE 22 - -CMD ["/usr/sbin/sshd", "-D"] http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/jenkins/gen_doc.sh ---------------------------------------------------------------------- diff --git a/tool/jenkins/gen_doc.sh b/tool/jenkins/gen_doc.sh new file mode 100644 index 0000000..cb78f0d --- /dev/null +++ b/tool/jenkins/gen_doc.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +#/** +# * +# * Licensed to the Apache Software Foundation (ASF) under one +# * or more contributor license agreements. See the NOTICE file +# * distributed with this work for additional information +# * regarding copyright ownership. The ASF licenses this file +# * to you under the Apache License, Version 2.0 (the +# * "License"); you may not use this file except in compliance +# * with the License. You may obtain a copy of the License at +# * +# * http://www.apache.org/licenses/LICENSE-2.0 +# * +# * Unless required by applicable law or agreed to in writing, software +# * distributed under the License is distributed on an "AS IS" BASIS, +# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# * See the License for the specific language governing permissions and +# * limitations under the License. +# */ + +# This script is used by Jenkins to update SINGA website + +echo Install PySinga, generate HTML files and update Singa website + +conda update singa +COMMIT=`git rev-parse --short HEAD` +cd doc +# generate the html files +./build.sh html +# checkout the current website files +svn co https://svn.apache.org/repos/asf/incubator/singa/site/trunk +# overwrite the existing files +cp -r _build/html/* trunk/ +# track newly added files and commit +cd trunk +svn add --force * --auto-props --parents --depth infinity -q +svn commit -m "update the docs by jenkins for commit $COMMIT" http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/jenkins/jenkins_doc.sh ---------------------------------------------------------------------- diff --git a/tool/jenkins/jenkins_doc.sh b/tool/jenkins/jenkins_doc.sh deleted file mode 100644 index e88daf2..0000000 --- a/tool/jenkins/jenkins_doc.sh +++ /dev/null @@ -1,37 +0,0 @@ -#!/usr/bin/env bash -#/** -# * -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ - -# This script is used by Jenkins to update SINGA website - -echo Install PySINGA, generate HTML files and update SINGA website -# pip install --upgrade http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/linux/latest/ubuntu16.04-cpp/singa-1.1.0-py2-none-any.whl -conda update singa -COMMIT=`git rev-parse --short HEAD` -cd doc -# generate the html files -./build.sh html -# checkout the current website files -svn co https://svn.apache.org/repos/asf/incubator/singa/site/trunk -# overwrite the existing files -cp -r _build/html/* trunk/ -# track newly added files and commit -cd trunk -svn add --force * --auto-props --parents --depth infinity -q -svn commit -m "update the docs by jenkins for commit $COMMIT" http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/jenkins/jenkins_test.sh ---------------------------------------------------------------------- diff --git a/tool/jenkins/jenkins_test.sh b/tool/jenkins/jenkins_test.sh deleted file mode 100644 index 0c16e2a..0000000 --- a/tool/jenkins/jenkins_test.sh +++ /dev/null @@ -1,57 +0,0 @@ -#!/usr/bin/env bash -#/** -# * -# * Licensed to the Apache Software Foundation (ASF) under one -# * or more contributor license agreements. See the NOTICE file -# * distributed with this work for additional information -# * regarding copyright ownership. The ASF licenses this file -# * to you under the Apache License, Version 2.0 (the -# * "License"); you may not use this file except in compliance -# * with the License. You may obtain a copy of the License at -# * -# * http://www.apache.org/licenses/LICENSE-2.0 -# * -# * Unless required by applicable law or agreed to in writing, software -# * distributed under the License is distributed on an "AS IS" BASIS, -# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# * See the License for the specific language governing permissions and -# * limitations under the License. -# */ - -# This script is used by Jenkins to compile and test SINGA - -echo Compile and test SINGA... -echo parameters: $1 -echo workspace: `pwd` -echo OS version: `cat /etc/issue` -echo kernal version: `uname -a` -echo CUDA version: $CUDA_VERSION -echo CUDNN version: $CUDNN_VERSION -COMMIT=`git rev-parse --short HEAD` -echo COMMIT HASH: $COMMIT -# set parameters -CUDA="OFF" -CUDNN="OFF" -if [ $1 = "CUDA" ]; then - CUDA="ON" - CUDNN="ON" -fi - -# setup env -rm -rf build -mkdir build - -if [ `uname` = "Darwin" ]; then - EXTRA_ARGS="-DPYTHON_LIBRARY=`python-config --prefix`/lib/libpython2.7.dylib -DPYTHON_INCLUDE_DIR=`python-config --prefix`/include/python2.7/" -fi - -# compile singa c++ -cd build -cmake -DUSE_CUDA=$CUDA -DENABLE_TEST=ON $EXTRA_ARGS ../ -make -# unit test cpp code -./bin/test_singa --gtest_output=xml:./gtest.xml -# unit test python code -cd ../test/python -PYTHONPATH=../../build/python/ python run.py -echo Job finished... http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/98e9d2f6/tool/jenkins/test.sh ---------------------------------------------------------------------- diff --git a/tool/jenkins/test.sh b/tool/jenkins/test.sh new file mode 100644 index 0000000..765fde1 --- /dev/null +++ b/tool/jenkins/test.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +#/** +# * +# * Licensed to the Apache Software Foundation (ASF) under one +# * or more contributor license agreements. See the NOTICE file +# * distributed with this work for additional information +# * regarding copyright ownership. The ASF licenses this file +# * to you under the Apache License, Version 2.0 (the +# * "License"); you may not use this file except in compliance +# * with the License. You may obtain a copy of the License at +# * +# * http://www.apache.org/licenses/LICENSE-2.0 +# * +# * Unless required by applicable law or agreed to in writing, software +# * distributed under the License is distributed on an "AS IS" BASIS, +# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# * See the License for the specific language governing permissions and +# * limitations under the License. +# */ + +# This script is used by Jenkins to compile and test Singa + +echo Compile and test Singa... +echo workspace: `pwd` +echo OS version: `cat /etc/issue` +echo kernal version: `uname -a` +COMMIT=`git rev-parse --short HEAD` +echo COMMIT HASH: $COMMIT +# set parameters +CUDA="OFF" +CUDNN="OFF" +if [ -z ${CUDNN_PATH+x} ]; then + CUDA="ON" + CUDNN="ON" + export CMAKE_PREFIX_PATH=$CUDNN_PATH:$CMAKE_PREFIX_PATH + export CMAKE_INCLUDE_PATH=$CUDNN_PATH/include:$CMAKE_INCLUDE_PATH + export CMAKE_LIBRARY_PATH=$CUDNN_PATH/lib64:$CMAKE_LIBRARY_PATH +fi + +# setup env +rm -rf build +mkdir build + +# TODO(wangwei) test python 3 according to env variable PY3K + +if [ `uname` = "Darwin" ]; then + EXTRA_ARGS="-DPYTHON_LIBRARY=`python-config --prefix`/lib/libpython2.7.dylib -DPYTHON_INCLUDE_DIR=`python-config --prefix`/include/python2.7/" +fi + +# compile c++ code +cd build +cmake -DUSE_CUDA=$CUDA -DENABLE_TEST=ON $EXTRA_ARGS ../ +make +# unit test cpp code +./bin/test_singa --gtest_output=xml:./gtest.xml +# unit test python code +cd ../test/python +PYTHONPATH=../../build/python/ python run.py +echo Job finished...
