Repository: incubator-singa Updated Branches: refs/heads/master 3f23c0d42 -> 330c87920
SINGA-288 - Publish wheel of PySINGA generated by Jenkins to public servers Update dockefiles to pass the OS_VERSION as an environment variable, which would be used to create the wheel file path. Update the jenkins_wheel.sh to move the whl file into $build_id/$commit_hash/$os_lang/ folder, and package this folder into $build_id.tar.gz. Update the README.md file with instructions of configuring the Jenkins to use Publish over SSH to copy $build_id.tar.gz to the public server. add build id to the prefix of artifact Project: http://git-wip-us.apache.org/repos/asf/incubator-singa/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-singa/commit/330c8792 Tree: http://git-wip-us.apache.org/repos/asf/incubator-singa/tree/330c8792 Diff: http://git-wip-us.apache.org/repos/asf/incubator-singa/diff/330c8792 Branch: refs/heads/master Commit: 330c87920057082fda2c35f664578bea97dbca91 Parents: 761c463 Author: wangwei <[email protected]> Authored: Fri Dec 23 15:44:18 2016 +0800 Committer: wangwei <[email protected]> Committed: Sat Dec 24 09:33:27 2016 +0800 ---------------------------------------------------------------------- tool/jenkins/README.md | 24 +++++++++++++++++++++--- tool/jenkins/docker/ubuntu14.04/Dockerfile | 9 ++++----- tool/jenkins/docker/ubuntu16.04/Dockerfile | 2 +- tool/jenkins/jenkins_wheel.sh | 8 ++++---- 4 files changed, 30 insertions(+), 13 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/330c8792/tool/jenkins/README.md ---------------------------------------------------------------------- diff --git a/tool/jenkins/README.md b/tool/jenkins/README.md index 1b5fb7c..b93b221 100644 --- a/tool/jenkins/README.md +++ b/tool/jenkins/README.md @@ -10,12 +10,13 @@ Those built binaries need to be archived for users to download. ## Install Jenkins [Jenkins Official Wiki](https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins) +The slave nodes for running different building environments are configured under 'Manage Jenkins'->'Manage nodes'. ## Configure Jenkins Multi-configuration Project Create a multi-configuration project and configure project as follows: ### Description - This job automatically pulls latest commits from apache SINGA github repository. + This job automatically pulls latest commits from SINGA github repository. It compiles and tests SINGA in different environments and creates PySINGA wheel distribution accordingly. ### General @@ -37,12 +38,29 @@ Create a multi-configuration project and configure project as follows: * Execute shell - command - ``bash -ex tool/jenkins/jenkins_wheel.sh $lang`` ### Post-build Actions - * Archive the artifacts - ``build/python/dist/**.whl`` - * Send build artifacts over SSH - source files ``build/python/dist/*.tar.gz``, Remove prefix `build/python/dist`, Exec command `tar xf *.tar.gz && rm *.tar.gz` * Publish JUnit test result report - Test report XMLs - ``**/gtest.xml, **/unittest.xml`` + * Archive the artifacts - ``build/python/dist/**.whl`` + * Send build artifacts over SSH + * In jenkins_wheel.sh, package the .whl file into $BUILD_ID.tar.gz. Inside the tar file, + the folder layout is `build_id/commit_hash/os_lang/*.whl`, where `os_lang` is the combination of os version, device programming language (cuda/cpp/opencl) and cudnn version. + * In `Manage Jenkins`-`Configure System`, configure the SSH for connecting to the remote public server and set the target folder location + * Source files - `build/python/dist/*.tar.gz` + * Remove prefix - `build/python/dist` + * Exec a command on the remote server to decompress the package and add a symlink to the latest build. E.g., on a Solaris server the command is + + cd <target_folder> && gunzip $BUILD_ID.tar.gz && tar xf $BUILD_ID.tar && chmod -R 755 $BUILD_ID && /bin/rm -f $BUILD_ID.tar && /bin/rm -f latest && ln -s $BUILD_ID/* latest + + * The file links on the remote public server would be like + + 32/84d56b7/ubuntu16.04-cpp/singa-1.0.1-py2-none-any.whl + 32/84d56b7/ubuntu16.04-cuda8.0-cudnn5/singa-1.0.1-py2-none-any.whl + ## Docker Images We provide in `docker` a number of singa docker images for Jenkins to use as slaves. +To run the docker images, + + nvidia-docker run --name <jenkins-slaveXX> -d <Image ID> ## Access Control Use `Role Strategy Plugin` to give read access for anonymous users. http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/330c8792/tool/jenkins/docker/ubuntu14.04/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/jenkins/docker/ubuntu14.04/Dockerfile b/tool/jenkins/docker/ubuntu14.04/Dockerfile index 6d96983..838be4a 100644 --- a/tool/jenkins/docker/ubuntu14.04/Dockerfile +++ b/tool/jenkins/docker/ubuntu14.04/Dockerfile @@ -4,14 +4,14 @@ # FROM nvidia/cuda:7.5-cudnn5-devel # FROM nvidia/cuda:7.5-cudnn4-devel # FROM nvidia/cuda:7.0-cudnn4-devel -FROM nvidia/cuda:8.0-cudnn5-devel +FROM nvidia/cuda:7.5-cudnn5-devel # install dependencies RUN apt-get update \ - && apt-get install -y --no-install-recommends git g++ cmake libprotobuf-dev libopenblas-dev protobuf-compiler python-dev python-pip wget\ + && apt-get install -y --no-install-recommends git build-essential autoconf libtool cmake libpcre3-dev libprotobuf-dev libopenblas-dev protobuf-compiler python-dev python-pip wget openssh-server\ && apt-get clean && apt-get autoremove && apt-get autoclean \ && rm -rf /var/lib/apt/lists/* \ - && pip --no-cache-dir install -U pip wheel numpy setuptools unittest-xml-reporting protobuf>=2.5.0,<3 + && pip install -U pip wheel numpy setuptools unittest-xml-reporting protobuf # install swig 3.0 @@ -28,8 +28,7 @@ ENV OS_VERSION ubuntu14.04 # download singa source RUN git clone https://github.com/apache/incubator-singa.git -# install ssh service -RUN apt-get install -y openssh-server +# config ssh service RUN mkdir /var/run/sshd RUN echo 'root:singa' | chpasswd RUN sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/330c8792/tool/jenkins/docker/ubuntu16.04/Dockerfile ---------------------------------------------------------------------- diff --git a/tool/jenkins/docker/ubuntu16.04/Dockerfile b/tool/jenkins/docker/ubuntu16.04/Dockerfile index 1a9434c..da8836e 100644 --- a/tool/jenkins/docker/ubuntu16.04/Dockerfile +++ b/tool/jenkins/docker/ubuntu16.04/Dockerfile @@ -12,7 +12,7 @@ RUN apt-get update \ && apt-get install -y --no-install-recommends git g++ cmake libprotobuf-dev libopenblas-dev protobuf-compiler python-dev python-pip swig wget openssh-server\ && apt-get clean && apt-get autoremove && apt-get autoclean \ && rm -rf /var/lib/apt/lists/* \ - && pip --no-cache-dir install -U pip wheel numpy setuptools unittest-xml-reporting protobuf>=2.5.0,<3 + && pip --no-cache-dir install -U pip wheel numpy setuptools unittest-xml-reporting protobuf # set environment ENV CPLUS_INCLUDE_PATH /usr/local/lib/python2.7/dist-packages/numpy/core/include:${CPLUS_INCLUDE_PATH} http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/330c8792/tool/jenkins/jenkins_wheel.sh ---------------------------------------------------------------------- diff --git a/tool/jenkins/jenkins_wheel.sh b/tool/jenkins/jenkins_wheel.sh index f131f68..1c8af27 100644 --- a/tool/jenkins/jenkins_wheel.sh +++ b/tool/jenkins/jenkins_wheel.sh @@ -28,16 +28,16 @@ echo kernal version: `uname -a` echo CUDA version: $CUDA_VERSION echo CUDNN version: $CUDNN_VERSION echo OS name: $OS_VERSION -COMMIT=`git rev-parse HEAD` +COMMIT=`git rev-parse --short HEAD` echo COMMIT HASH: $COMMIT # set parameters CUDA="OFF" CUDNN="OFF" -FOLDER=$COMMIT/$OS_VERSION-cpp/ +FOLDER=$BUILD_NUMBER/$COMMIT/$OS_VERSION-cpp if [ $1 = "CUDA" ]; then CUDA="ON" CUDNN="ON" - FOLDER=$COMMIT/$OS_VERSION-cuda$CUDA_VERSION-cudnn$CUDNN_VERSION/ + FOLDER=$BUILD_NUMBER/$COMMIT/$OS_VERSION-cuda$CUDA_VERSION-cudnn$CUDNN_VERSION fi echo wheel file folder: build/python/dist/whl/$FOLDER @@ -58,7 +58,7 @@ python setup.py bdist_wheel cd dist mkdir -p $FOLDER mv *.whl $FOLDER/ -tar czf $FOLDER.tar.gz $FOLDER/* +tar czf $BUILD_NUMBER.tar.gz $FOLDER/* # unit test python code cd ../../../test/python
