SINGA-303 Create conda packages Fix bugs: _singa_wrap must be ended with .so (cannot use dylib) on osx must link with libs from conda, there are two approaches 1. export CMAKE_PREFIX_PATH=$PREFIX 2. pass each dep lib path to cmake. Otherwise the system lib (e.g., python) path would be linked into _singa_wrap.so.
@rpath/libpython2.7.dylib is replaced by a path RPATH list that has libpython2.7.dylib. conda would store an entry @load_path/../../.. in RPATH for _singa_wrap.so on osx. @load_path is the path of _singa_wrap.so. update readme Project: http://git-wip-us.apache.org/repos/asf/incubator-singa/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-singa/commit/ca0be884 Tree: http://git-wip-us.apache.org/repos/asf/incubator-singa/tree/ca0be884 Diff: http://git-wip-us.apache.org/repos/asf/incubator-singa/diff/ca0be884 Branch: refs/heads/master Commit: ca0be884c5cedc83fa10aab1547e9453c006a264 Parents: 493fc65 Author: xiezl <[email protected]> Authored: Fri Feb 24 10:22:56 2017 +0800 Committer: wangwei <[email protected]> Committed: Fri Feb 24 23:56:25 2017 +0800 ---------------------------------------------------------------------- python/CMakeLists.txt | 8 ++++++++ tool/conda/build.sh | 4 ++++ tool/jenkins/README.md | 29 +++++++++++++++-------------- 3 files changed, 27 insertions(+), 14 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/ca0be884/python/CMakeLists.txt ---------------------------------------------------------------------- diff --git a/python/CMakeLists.txt b/python/CMakeLists.txt index 7445a57..177326e 100644 --- a/python/CMakeLists.txt +++ b/python/CMakeLists.txt @@ -129,3 +129,11 @@ CONFIGURE_FILE(${SETUP_PY_IN} ${SETUP_PY}) #create python/singa/proto/__init__.py FILE(WRITE ${CMAKE_BINARY_DIR}/python/singa/proto/__init__.py "") +IF(APPLE) + ADD_CUSTOM_TARGET( + change_suffix ALL + COMMAND ${CMAKE_COMMAND} -E rename "${CMAKE_BINARY_DIR}/python/singa/_singa_wrap.dylib" "${CMAKE_BINARY_DIR}/python/singa/_singa_wrap.so" + COMMENT "change .dylib to .so in mac system" + ) + ADD_DEPENDENCIES(change_suffix _singa_wrap) +ENDIF(APPLE) http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/ca0be884/tool/conda/build.sh ---------------------------------------------------------------------- diff --git a/tool/conda/build.sh b/tool/conda/build.sh index f3a84d9..926da71 100644 --- a/tool/conda/build.sh +++ b/tool/conda/build.sh @@ -15,8 +15,12 @@ # limitations under the License. # +# to compile swig api files which depdend on numpy.i export export CPLUS_INCLUDE_PATH=`python -c "import numpy; print numpy.get_include()"`:$CPLUS_INCLUDE_PATH +# to let cmake use the dependent libs installed by conda, including python +export CMAKE_PREFIX_PATH=$PREFIX + mkdir build cd build USE_CUDA=ON http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/ca0be884/tool/jenkins/README.md ---------------------------------------------------------------------- diff --git a/tool/jenkins/README.md b/tool/jenkins/README.md index 6a1cb1e..9a76ee8 100644 --- a/tool/jenkins/README.md +++ b/tool/jenkins/README.md @@ -1,12 +1,11 @@ # Jenkins CI Support ## Introduction -This documentation is to guide SINGA developers to setup Jenkins service. +This documentation is to guide SINGA developers to setup Jenkins service to support continuous integration on GPU systems. After each commit, +1. SINGA should be compiled and tested automatically under different settings (e.g., OS and hardware). +2. Convenient binaries should be generated automatically and archived. -We use Jenkins to support continuous integration. -After each commit, we want to automatically compile and test SINGA -under different OS and settings. -Those built binaries need to be archived for users to download. +Continuous integration for CPU systems is enabled via [Travis](../travis). ## Install Jenkins [Jenkins Official Wiki](https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins) @@ -56,18 +55,21 @@ Each node should configure the following environment variable * create conda package Execute shell - command - - git push --mirror <https://username:[email protected]/mirror_repo> - /root/miniconda/conda tool/conda/ + git push https://username:[email protected]/nusdbsystem/incubator-singa.git -f + bash -ex tool/jenkins/jenkins_test.sh $lang export CONDA_BLD_PATH=/root/conda-bld-$BUILD_NUMBER mkdir $CONDA_BLD_PATH - /root/miniconda/anaconda -t $ANACONDA_UPLOAD_TOKEN upload -u $USER -l main $CONDA_BLD_PATH/$OS/singa-*.tar.bz2 --force + /root/miniconda/bin/conda-build tool/conda + /root/miniconda/bin/anaconda -t ANACONDA_UPLOAD_TOKEN upload -u nusdbsystem -l main $CONDA_BLD_PATH/linux-64/singa-*.tar.bz2 --force + It first pushes to a mirror site to invoke travis-ci for CPU package creation; - Then it creates the conda package for GPU and uploads the package. + Then it compiles and runs unit tests; + Finally it creates the conda package for GPU and upload it. ### Post-build Actions * Publish JUnit test result report - Test report XMLs - ``**/gtest.xml, **/unittest.xml`` - * (optional) Archive the artifacts - ``build/python/dist/**.whl, build/debian/**.deb`` + * (optional) Archive the artifacts - ``build/debian/**.deb`` * Send build artifacts (Debian package) over SSH for wheel * ../debian/build.sh packages the .deb file into $BUILD_ID.tar.gz. Inside the tar file, the folder layout is `build_id/commit_hash/os_lang/*.deb`, where `os_lang` is the combination of os version, device programming language (cuda/cpp/opencl) and cudnn version. @@ -81,12 +83,11 @@ Each node should configure the following environment variable * The file links on the remote public server would be like - debian/32/84d56b7/ubuntu16.04-cpp/singa-1.0.1.deb - debian/32/84d56b7/ubuntu16.04-cuda8.0-cudnn5/singa-1.0.1.deb - + debian/32/84d56b7/ubuntu14.04-cpp/singa-1.0.1.deb + debian/32/84d56b7/ubuntu14.04-cuda8.0-cudnn5/singa-1.0.1.deb ### Docker Images -We provide in a number of singa docker [images](../docker) for Jenkins to use as slaves. +We provide in a number of singa docker [images](./docker) for Jenkins to use as slaves. To run the docker images, nvidia-docker run --name <jenkins-slaveXX> -d <Image ID>
