This is an automated email from the ASF dual-hosted git repository. pono pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push: new 7484437 [Do not merge] Move to new CI (#8960) 7484437 is described below commit 7484437d21c5e1456c24bff31e6642cf0a47db4d Author: Marco de Abreu <marcoab...@users.noreply.github.com> AuthorDate: Wed Dec 6 19:31:19 2017 +0100 [Do not merge] Move to new CI (#8960) * Squashed commit of the following: commit ce53ad38a6ef260c7a5870faa58583b5fa2b1e30 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Sat Nov 25 10:53:09 2017 +0100 remove whitespace commit 71f1f8d5a71dd495a12594a4f01bf275fbded186 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Sat Nov 25 10:37:04 2017 +0100 Decrease timeout in Jenkinsfile commit ef88da18dd0f29b28ae15bb9840f3ff393e4e77f Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Sat Nov 25 10:35:44 2017 +0100 Cleanup and add comments commit ad59a48e4f68fe855029ba23a78bf9fa1abf1544 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Sat Nov 25 09:26:51 2017 +0100 Fix Integration Test Caffe-GPU to work on ubuntu16.04 with CUDA 8 commit fc7cf23e92a6bf48cc36968df274e55c4d47a17a Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Sat Nov 25 07:42:49 2017 +0100 Add verbose output to perl tests commit f076312de899fe859df3d03e74b3f97923c96a2c Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Sat Nov 25 06:40:31 2017 +0100 Adjust Jenkinsfile to ensure integration stage being run on nvidia-docker commit 9ed3cd0ba7411ad686bb0162e473684c7f689211 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Sat Nov 25 02:14:54 2017 +0100 Add nvidia dependencies to caffe_gpu commit 103225b70f734c92aa69416aa94eff3b24b28c5a Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Fri Nov 24 17:48:32 2017 +0100 Update GPU containers to ubuntu16.04 / CUDA 8 to fix OpenBLAS-hang 2 commit d1a12065826dabfbbe2102b375fb22a794eb5b3d Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Fri Nov 24 17:22:51 2017 +0100 Update GPU containers to ubuntu16.04 / CUDA 8 to fix OpenBLAS-hang commit b193874f0fd33a2d985fe430b9787d5cbf479662 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Fri Nov 24 14:39:36 2017 +0100 Fix typo commit c1e303468f494e28b8a4c625434a66c9b2c72282 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Fri Nov 24 14:25:43 2017 +0100 Fix typo commit 143fbe258e067d59efcf08ac948850b6de4cd9a0 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Fri Nov 24 14:18:44 2017 +0100 Fix unittest due to cuda commit f7255519bc96dcd25bfe83b89f8ec506af1caba6 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Fri Nov 24 14:00:48 2017 +0100 Make sure containers are run inside the right docker binary commit 7ae2c67ad91fcc2a7093f4d1bf417e6096bfa32e Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Fri Nov 24 13:43:56 2017 +0100 Fix typo commit 26d3a247ed502ea6f3ef25ded63ebe3f62ed9c72 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Fri Nov 24 13:42:34 2017 +0100 Trigger build commit aca7005ddb91ad9ce68fd4ee471664486e44a3fd Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Fri Nov 24 12:49:36 2017 +0100 Separated GPU-Build dockerfiles from runtime Dockerfiles commit 6091e5b171e3c50cecdfeacce04d21e2b3f65693 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Fri Nov 24 03:43:12 2017 +0100 Enable install of nvidia drivers in GPU Dockerfiles commit f0a3fe7474fb13d96bc84671e3402a8b33e71ccd Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Fri Nov 24 02:31:58 2017 +0100 Add cuda stub lib path to Makefile to fix libcuda.so.1 not found error commit 8233a3d032cdf77142f9e9c0424134da390f36fa Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Fri Nov 24 02:25:29 2017 +0100 Add softlink to fix libcuda.so.1 not found-error commit 8c3e50ae98a7de95476bc15dd8ca8c146003c1f1 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Fri Nov 24 01:44:49 2017 +0100 Fix libcuda.so.1 not found during build commit 2dbf35424060ec36d8cd4a084ddb3f1a976323cd Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Thu Nov 23 00:12:58 2017 +0100 Re-enable DEV-Flag in Jenkinsfile commit 6083f109c0811c310b7f22268c5d6556badf4426 Author: Pedro Larroy <pllar...@amazon.com> Date: Wed Nov 22 14:38:37 2017 -0800 Fix uninitialized array warning commit 6e0dd21c121b275b341dbb9c304ce73ded64eae6 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Wed Nov 22 22:49:55 2017 +0100 Support CUDA-Builds on non-gpu-instances commit 5f53708e369c41ed77e76fd6623a9db6573f1b62 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Wed Nov 22 15:15:36 2017 +0100 Temp. disable dev compile mode to prevent warnings being treated as errors commit 5e5dff50af43ed7d618122d1c2f0379a1d7193fc Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Wed Nov 22 14:10:22 2017 +0100 Fix typo in Jenkins for windows commit a3d9f7483a693b0c513c677baaa5ba6621e722d1 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Wed Nov 22 14:07:18 2017 +0100 Fix tests/ci_build/with_the_same_user: line 27: /etc/sudoers.d/90-nopasswd-sudo: No such file or directory commit d76e342cfb76361ac166bc5e770bbef7721317d5 Merge: 5762ef88 0c0a5626 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Wed Nov 22 14:07:04 2017 +0100 Merge branch 'v0.12.0' of github.com:MXNetEdge/mabreu-incubator-mxnet into v0.12.0 commit 5762ef88b85754809fc98a01b149be5036737b64 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Wed Nov 22 13:46:57 2017 +0100 Add Kellens fix for C5 commit a93a9cd74ec362b4e3417e840169e3ce8b70cfba Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Wed Nov 22 11:40:12 2017 +0100 Add Jenkinsfile for windows commit 0c0a5626a614ac203cc7e93fbd5166e4f50267bb Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Wed Nov 22 11:40:12 2017 +0100 Add Jenkinsfile for windows commit 175d157171611778ee6a1e1804e4611e7ef4c136 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Mon Nov 20 19:56:18 2017 +0100 Remove failing test commit bb15df1464a02fbc9dff042b24e52b49187b9d55 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Mon Nov 20 17:25:29 2017 +0100 Disable unnecessary CUDA-archs commit c68265892256eb8756d2eba696ccc41766fbe3db Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Mon Nov 20 16:28:15 2017 +0100 Disable failing test test_operator:test_depthwise_convolution commit 07028474d10326029aa27afb81ff3eb3fa063b5f Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Mon Nov 20 15:01:08 2017 +0100 Increase git timeout commit 61d689424541801bad5a2ec5b46581810281d9a8 Author: Marco de Abreu <marco.g.abreu+git...@gmail.com> Date: Mon Nov 20 13:16:28 2017 +0100 Split up Jenkins-Tasks to appropriate machines * Remove temp file * Add separate compile job for MKLML-CPU * Fix typo * Revert Jenkinsfile * Remove MKL Experimental * Revert "Remove MKL Experimental" This reverts commit 5e0f20c67e872c1a5af7491610ab55d37f6fc5b8. * Disable test_gluon:test_dtype * Cleanup * Add sudo install to lint * Add comment about MKLML * Fix typo * Clean up dockerfiles regarding mklml * Enable failing tests * Restructure dockerfiles * Fix typo * Fix Dockerfiles * Update mklml library * MKLML fix * Try to fix failing MKLML-gpu build * Fix typo * Disable flaky test https://github.com/apache/incubator-mxnet/issues/8892 * Remove workspace from jenkinsfile * Revert "Remove workspace from jenkinsfile" This reverts commit c2cfe5db296eafb008f63a7ddb0c841ac3b8313e. * Misc changes Merge Jenkinsfiles Documented Dockerfiles and scripts Cleaned up test comments Restore Dockerfile.ubuntu1404_cuda75_cudnn5 * fix typo * Fix typo * Revert KNOWN_CUDA_ARCH changes * Change makefile comment * Remove Jenkinsfile_windows * Add myself to CONTRIBUTORS.md * Cleanup * Add CUDA_ARCH to Jenkinsfile * Fix typo * Fix typo * Fix typo * Fix typo * Add cuda_archs to ci_build * Add cuda_archs to ci_build * Fix typo and comment * Disable flaky test test_loss:test_ctc_loss_train #8892 --- CONTRIBUTORS.md | 2 + Jenkinsfile | 353 +++++++++++---------- Makefile | 4 + perl-package/test.sh | 2 +- ...{Dockerfile.mklml_gpu => Dockerfile.build_cuda} | 15 +- tests/ci_build/Dockerfile.caffe_gpu | 13 +- tests/ci_build/Dockerfile.cpu | 2 +- .../{Dockerfile.mklml_gpu => Dockerfile.cpu_mklml} | 11 +- tests/ci_build/Dockerfile.gpu | 4 +- .../{Dockerfile.mklml_gpu => Dockerfile.gpu_mklml} | 7 +- tests/ci_build/Dockerfile.lint | 5 +- tests/ci_build/ci_build.sh | 21 +- tests/ci_build/install/ubuntu_install_core.sh | 5 +- ...tu_install_core.sh => ubuntu_install_nvidia.sh} | 15 +- tests/ci_build/pip_tests/Dockerfile.in.pip_cpu | 2 +- tests/python/unittest/test_loss.py | 4 +- 16 files changed, 271 insertions(+), 194 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 7209b7c..1c42e03 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -150,3 +150,5 @@ List of Contributors * [Manu Seth](https://github.com/mseth10/) * [Calum Leslie](https://github.com/calumleslie) * [Andre Tamm](https://github.com/andretamm) +* [Marco de Abreu](https://github.com/marcoabreu) + - Marco is the creator of the current MXNet CI. diff --git a/Jenkinsfile b/Jenkinsfile index cbe6375..c4c16ad 100644 --- a/Jenkinsfile +++ b/Jenkinsfile @@ -52,12 +52,12 @@ def init_git_win() { def make(docker_type, make_flag) { timeout(time: max_time, unit: 'MINUTES') { try { - sh "${docker_run} ${docker_type} make ${make_flag}" + sh "${docker_run} ${docker_type} --dockerbinary docker make ${make_flag}" } catch (exc) { echo 'Incremental compilation failed with ${exc}. Fall back to build from scratch' - sh "${docker_run} ${docker_type} sudo make clean" - sh "${docker_run} ${docker_type} sudo make -C amalgamation/ clean" - sh "${docker_run} ${docker_type} make ${make_flag}" + sh "${docker_run} ${docker_type} --dockerbinary docker sudo make clean" + sh "${docker_run} ${docker_type} --dockerbinary docker sudo make -C amalgamation/ clean" + sh "${docker_run} ${docker_type} --dockerbinary docker make ${make_flag}" } } } @@ -85,17 +85,17 @@ echo ${libs} | sed -e 's/,/ /g' | xargs md5sum // Python 2 def python2_ut(docker_type) { timeout(time: max_time, unit: 'MINUTES') { - sh "${docker_run} ${docker_type} find . -name '*.pyc' -type f -delete" - sh "${docker_run} ${docker_type} PYTHONPATH=./python/ nosetests-2.7 --with-timer --verbose tests/python/unittest" - sh "${docker_run} ${docker_type} PYTHONPATH=./python/ nosetests-2.7 --with-timer --verbose tests/python/train" + sh "${docker_run} ${docker_type} --dockerbinary docker find . -name '*.pyc' -type f -delete" + sh "${docker_run} ${docker_type} --dockerbinary docker PYTHONPATH=./python/ nosetests-2.7 --with-timer --verbose tests/python/unittest" + sh "${docker_run} ${docker_type} --dockerbinary docker PYTHONPATH=./python/ nosetests-2.7 --with-timer --verbose tests/python/train" } } // Python 3 def python3_ut(docker_type) { timeout(time: max_time, unit: 'MINUTES') { - sh "${docker_run} ${docker_type} find . -name '*.pyc' -type f -delete" - sh "${docker_run} ${docker_type} PYTHONPATH=./python/ nosetests-3.4 --with-timer --verbose tests/python/unittest" + sh "${docker_run} ${docker_type} --dockerbinary docker find . -name '*.pyc' -type f -delete" + sh "${docker_run} ${docker_type} --dockerbinary docker PYTHONPATH=./python/ nosetests-3.4 --with-timer --verbose tests/python/unittest" } } @@ -120,7 +120,7 @@ def python3_gpu_ut(docker_type) { try { stage("Sanity Check") { timeout(time: max_time, unit: 'MINUTES') { - node('mxnetlinux') { + node('mxnetlinux-cpu') { ws('workspace/sanity') { init_git() sh "python tools/license_header.py check" @@ -133,43 +133,82 @@ try { stage('Build') { parallel 'CPU: Openblas': { - node('mxnetlinux') { + node('mxnetlinux-cpu') { ws('workspace/build-cpu') { init_git() def flag = """ \ - DEV=1 \ - USE_PROFILER=1 \ - USE_CPP_PACKAGE=1 \ - USE_BLAS=openblas \ - -j\$(nproc) - """ + DEV=1 \ + USE_PROFILER=1 \ + USE_CPP_PACKAGE=1 \ + USE_BLAS=openblas \ + -j\$(nproc) + """ make("cpu", flag) pack_lib('cpu') } } }, - 'GPU: CUDA7.5+cuDNN5': { - node('mxnetlinux') { + 'CPU: MKLML': { + node('mxnetlinux-cpu') { + ws('workspace/build-mklml-cpu') { + init_git() + def flag = """ \ + DEV=1 \ + USE_PROFILER=1 \ + USE_CPP_PACKAGE=1 \ + USE_BLAS=openblas \ + USE_MKL2017=1 \ + USE_MKL2017_EXPERIMENTAL=1 \ + -j\$(nproc) + """ + make("cpu_mklml", flag) + pack_lib('mklml_cpu') + } + } + }, + 'GPU: MKLML': { + node('mxnetlinux-cpu') { + ws('workspace/build-mklml-gpu') { + init_git() + def flag = """ \ + DEV=1 \ + USE_PROFILER=1 \ + USE_CPP_PACKAGE=1 \ + USE_BLAS=openblas \ + USE_MKL2017=1 \ + USE_MKL2017_EXPERIMENTAL=1 \ + USE_CUDA=1 \ + USE_CUDA_PATH=/usr/local/cuda \ + USE_CUDNN=1 \ + -j\$(nproc) + """ + make("build_cuda", flag) + pack_lib('mklml_gpu') + } + } + }, + 'GPU: CUDA8.0+cuDNN5': { + node('mxnetlinux-cpu') { ws('workspace/build-gpu') { init_git() def flag = """ \ - DEV=1 \ - USE_PROFILER=1 \ - USE_BLAS=openblas \ - USE_CUDA=1 \ - USE_CUDA_PATH=/usr/local/cuda \ - USE_CUDNN=1 \ - USE_CPP_PACKAGE=1 \ - -j\$(nproc) - """ - make('gpu', flag) + DEV=1 \ + USE_PROFILER=1 \ + USE_BLAS=openblas \ + USE_CUDA=1 \ + USE_CUDA_PATH=/usr/local/cuda \ + USE_CUDNN=1 \ + USE_CPP_PACKAGE=1 \ + -j\$(nproc) + """ + make('build_cuda', flag) pack_lib('gpu') stash includes: 'build/cpp-package/example/test_score', name: 'cpp_test_score' } } }, 'Amalgamation MIN': { - node('mxnetlinux') { + node('mxnetlinux-cpu') { ws('workspace/amalgamationmin') { init_git() make('cpu', '-C amalgamation/ clean') @@ -178,7 +217,7 @@ try { } }, 'Amalgamation': { - node('mxnetlinux') { + node('mxnetlinux-cpu') { ws('workspace/amalgamation') { init_git() make('cpu', '-C amalgamation/ clean') @@ -186,93 +225,73 @@ try { } } }, - 'GPU: MKLML': { - node('mxnetlinux') { - ws('workspace/build-mklml') { - init_git() - def flag = """ \ - DEV=1 \ - USE_PROFILER=1 \ - USE_BLAS=openblas \ - USE_MKL2017=1 \ - USE_MKL2017_EXPERIMENTAL=1 \ - USE_CUDA=1 \ - USE_CUDA_PATH=/usr/local/cuda \ - USE_CUDNN=1 \ - USE_CPP_PACKAGE=1 \ - -j\$(nproc) - """ - make('mklml_gpu', flag) - pack_lib('mklml') - } - } - }, - 'CPU windows':{ - node('mxnetwindows') { + 'Build CPU windows':{ + node('mxnetwindows-cpu') { ws('workspace/build-cpu') { withEnv(['OpenBLAS_HOME=C:\\mxnet\\openblas', 'OpenCV_DIR=C:\\mxnet\\opencv_vc14', 'CUDA_PATH=C:\\CUDA\\v8.0']) { init_git_win() bat """mkdir build_vc14_cpu - call "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\x86_amd64\\vcvarsx86_amd64.bat" - cd build_vc14_cpu - cmake -G \"Visual Studio 14 2015 Win64\" -DUSE_CUDA=0 -DUSE_CUDNN=0 -DUSE_NVRTC=0 -DUSE_OPENCV=1 -DUSE_OPENMP=1 -DUSE_PROFILER=1 -DUSE_BLAS=open -DUSE_LAPACK=1 -DUSE_DIST_KVSTORE=0 ${env.WORKSPACE}""" + call "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\x86_amd64\\vcvarsx86_amd64.bat" + cd build_vc14_cpu + cmake -G \"Visual Studio 14 2015 Win64\" -DUSE_CUDA=0 -DUSE_CUDNN=0 -DUSE_NVRTC=0 -DUSE_OPENCV=1 -DUSE_OPENMP=1 -DUSE_PROFILER=1 -DUSE_BLAS=open -DUSE_LAPACK=1 -DUSE_DIST_KVSTORE=0 ${env.WORKSPACE}""" bat 'C:\\mxnet\\build_vc14_cpu.bat' bat '''rmdir /s/q pkg_vc14_cpu - mkdir pkg_vc14_cpu\\lib - mkdir pkg_vc14_cpu\\python - mkdir pkg_vc14_cpu\\include - mkdir pkg_vc14_cpu\\build - copy build_vc14_cpu\\Release\\libmxnet.lib pkg_vc14_cpu\\lib - copy build_vc14_cpu\\Release\\libmxnet.dll pkg_vc14_cpu\\build - xcopy python pkg_vc14_cpu\\python /E /I /Y - xcopy include pkg_vc14_cpu\\include /E /I /Y - xcopy dmlc-core\\include pkg_vc14_cpu\\include /E /I /Y - xcopy mshadow\\mshadow pkg_vc14_cpu\\include\\mshadow /E /I /Y - xcopy nnvm\\include pkg_vc14_cpu\\nnvm\\include /E /I /Y - del /Q *.7z - 7z.exe a vc14_cpu.7z pkg_vc14_cpu\\ - ''' + mkdir pkg_vc14_cpu\\lib + mkdir pkg_vc14_cpu\\python + mkdir pkg_vc14_cpu\\include + mkdir pkg_vc14_cpu\\build + copy build_vc14_cpu\\Release\\libmxnet.lib pkg_vc14_cpu\\lib + copy build_vc14_cpu\\Release\\libmxnet.dll pkg_vc14_cpu\\build + xcopy python pkg_vc14_cpu\\python /E /I /Y + xcopy include pkg_vc14_cpu\\include /E /I /Y + xcopy dmlc-core\\include pkg_vc14_cpu\\include /E /I /Y + xcopy mshadow\\mshadow pkg_vc14_cpu\\include\\mshadow /E /I /Y + xcopy nnvm\\include pkg_vc14_cpu\\nnvm\\include /E /I /Y + del /Q *.7z + 7z.exe a vc14_cpu.7z pkg_vc14_cpu\\ + ''' stash includes: 'vc14_cpu.7z', name: 'vc14_cpu' } } } }, - 'GPU windows':{ - node('mxnetwindows') { + //Todo: Set specific CUDA_ARCh for windows builds in cmake + 'Build GPU windows':{ + node('mxnetwindows-cpu') { ws('workspace/build-gpu') { withEnv(['OpenBLAS_HOME=C:\\mxnet\\openblas', 'OpenCV_DIR=C:\\mxnet\\opencv_vc14', 'CUDA_PATH=C:\\CUDA\\v8.0']) { - init_git_win() - bat """mkdir build_vc14_gpu - call "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\x86_amd64\\vcvarsx86_amd64.bat" - cd build_vc14_gpu - cmake -G \"NMake Makefiles JOM\" -DUSE_CUDA=1 -DUSE_CUDNN=1 -DUSE_NVRTC=1 -DUSE_OPENCV=1 -DUSE_OPENMP=1 -DUSE_PROFILER=1 -DUSE_BLAS=open -DUSE_LAPACK=1 -DUSE_DIST_KVSTORE=0 -DCUDA_ARCH_NAME=All -DCMAKE_CXX_FLAGS_RELEASE="/FS /MD /O2 /Ob2 /DNDEBUG" -DCMAKE_BUILD_TYPE=Release ${env.WORKSPACE}""" - bat 'C:\\mxnet\\build_vc14_gpu.bat' - bat '''rmdir /s/q pkg_vc14_gpu - mkdir pkg_vc14_gpu\\lib - mkdir pkg_vc14_gpu\\python - mkdir pkg_vc14_gpu\\include - mkdir pkg_vc14_gpu\\build - copy build_vc14_gpu\\libmxnet.lib pkg_vc14_gpu\\lib - copy build_vc14_gpu\\libmxnet.dll pkg_vc14_gpu\\build - xcopy python pkg_vc14_gpu\\python /E /I /Y - xcopy include pkg_vc14_gpu\\include /E /I /Y - xcopy dmlc-core\\include pkg_vc14_gpu\\include /E /I /Y - xcopy mshadow\\mshadow pkg_vc14_gpu\\include\\mshadow /E /I /Y - xcopy nnvm\\include pkg_vc14_gpu\\nnvm\\include /E /I /Y - del /Q *.7z - 7z.exe a vc14_gpu.7z pkg_vc14_gpu\\ - ''' - stash includes: 'vc14_gpu.7z', name: 'vc14_gpu' + init_git_win() + bat """mkdir build_vc14_gpu + call "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\x86_amd64\\vcvarsx86_amd64.bat" + cd build_vc14_gpu + cmake -G \"NMake Makefiles JOM\" -DUSE_CUDA=1 -DUSE_CUDNN=1 -DUSE_NVRTC=1 -DUSE_OPENCV=1 -DUSE_OPENMP=1 -DUSE_PROFILER=1 -DUSE_BLAS=open -DUSE_LAPACK=1 -DUSE_DIST_KVSTORE=0 -DCUDA_ARCH_NAME=All -DCMAKE_CXX_FLAGS_RELEASE="/FS /MD /O2 /Ob2 /DNDEBUG" -DCMAKE_BUILD_TYPE=Release ${env.WORKSPACE}""" + bat 'C:\\mxnet\\build_vc14_gpu.bat' + bat '''rmdir /s/q pkg_vc14_gpu + mkdir pkg_vc14_gpu\\lib + mkdir pkg_vc14_gpu\\python + mkdir pkg_vc14_gpu\\include + mkdir pkg_vc14_gpu\\build + copy build_vc14_gpu\\libmxnet.lib pkg_vc14_gpu\\lib + copy build_vc14_gpu\\libmxnet.dll pkg_vc14_gpu\\build + xcopy python pkg_vc14_gpu\\python /E /I /Y + xcopy include pkg_vc14_gpu\\include /E /I /Y + xcopy dmlc-core\\include pkg_vc14_gpu\\include /E /I /Y + xcopy mshadow\\mshadow pkg_vc14_gpu\\include\\mshadow /E /I /Y + xcopy nnvm\\include pkg_vc14_gpu\\nnvm\\include /E /I /Y + del /Q *.7z + 7z.exe a vc14_gpu.7z pkg_vc14_gpu\\ + ''' + stash includes: 'vc14_gpu.7z', name: 'vc14_gpu' } } } - } + } } stage('Unit Test') { parallel 'Python2: CPU': { - node('mxnetlinux') { + node('mxnetlinux-cpu') { ws('workspace/ut-python2-cpu') { init_git() unpack_lib('cpu') @@ -281,7 +300,7 @@ try { } }, 'Python3: CPU': { - node('mxnetlinux') { + node('mxnetlinux-cpu') { ws('workspace/ut-python3-cpu') { init_git() unpack_lib('cpu') @@ -290,7 +309,7 @@ try { } }, 'Python2: GPU': { - node('mxnetlinux') { + node('mxnetlinux-gpu') { ws('workspace/ut-python2-gpu') { init_git() unpack_lib('gpu', mx_lib) @@ -299,7 +318,7 @@ try { } }, 'Python3: GPU': { - node('mxnetlinux') { + node('mxnetlinux-gpu') { ws('workspace/ut-python3-gpu') { init_git() unpack_lib('gpu', mx_lib) @@ -308,43 +327,43 @@ try { } }, 'Python2: MKLML-CPU': { - node('mxnetlinux') { + node('mxnetlinux-cpu') { ws('workspace/ut-python2-mklml-cpu') { init_git() - unpack_lib('mklml') - python2_ut('mklml_gpu') + unpack_lib('mklml_cpu') + python2_ut('cpu_mklml') } } }, 'Python2: MKLML-GPU': { - node('mxnetlinux') { + node('mxnetlinux-gpu') { ws('workspace/ut-python2-mklml-gpu') { init_git() - unpack_lib('mklml') - python2_gpu_ut('mklml_gpu') + unpack_lib('mklml_gpu') + python2_gpu_ut('gpu_mklml') } } }, 'Python3: MKLML-CPU': { - node('mxnetlinux') { + node('mxnetlinux-cpu') { ws('workspace/ut-python3-mklml-cpu') { init_git() - unpack_lib('mklml') - python3_ut('mklml_gpu') + unpack_lib('mklml_cpu') + python3_ut('cpu_mklml') } } }, 'Python3: MKLML-GPU': { - node('mxnetlinux') { + node('mxnetlinux-gpu') { ws('workspace/ut-python3-mklml-gpu') { init_git() - unpack_lib('mklml') - python3_gpu_ut('mklml_gpu') + unpack_lib('mklml_gpu') + python3_gpu_ut('gpu_mklml') } } }, 'Scala: CPU': { - node('mxnetlinux') { + node('mxnetlinux-cpu') { ws('workspace/ut-scala-cpu') { init_git() unpack_lib('cpu') @@ -356,7 +375,7 @@ try { } }, 'Perl: CPU': { - node('mxnetlinux') { + node('mxnetlinux-cpu') { ws('workspace/ut-perl-cpu') { init_git() unpack_lib('cpu') @@ -367,7 +386,7 @@ try { } }, 'Perl: GPU': { - node('mxnetlinux') { + node('mxnetlinux-gpu') { ws('workspace/ut-perl-gpu') { init_git() unpack_lib('gpu') @@ -378,7 +397,7 @@ try { } }, 'R: CPU': { - node('mxnetlinux') { + node('mxnetlinux-cpu') { ws('workspace/ut-r-cpu') { init_git() unpack_lib('cpu') @@ -393,7 +412,7 @@ try { } }, 'R: GPU': { - node('mxnetlinux') { + node('mxnetlinux-gpu') { ws('workspace/ut-r-gpu') { init_git() unpack_lib('gpu') @@ -408,102 +427,102 @@ try { } }, 'Python 2: CPU Win':{ - node('mxnetwindows') { + node('mxnetwindows-cpu') { ws('workspace/ut-python-cpu') { init_git_win() unstash 'vc14_cpu' bat '''rmdir /s/q pkg_vc14_cpu - 7z x -y vc14_cpu.7z''' + 7z x -y vc14_cpu.7z''' bat """xcopy C:\\mxnet\\data data /E /I /Y - xcopy C:\\mxnet\\model model /E /I /Y - call activate py2 - set PYTHONPATH=${env.WORKSPACE}\\pkg_vc14_cpu\\python - del /S /Q ${env.WORKSPACE}\\pkg_vc14_cpu\\python\\*.pyc - C:\\mxnet\\test_cpu.bat""" + xcopy C:\\mxnet\\model model /E /I /Y + call activate py2 + set PYTHONPATH=${env.WORKSPACE}\\pkg_vc14_cpu\\python + del /S /Q ${env.WORKSPACE}\\pkg_vc14_cpu\\python\\*.pyc + C:\\mxnet\\test_cpu.bat""" } } }, 'Python 3: CPU Win': { - node('mxnetwindows') { + node('mxnetwindows-cpu') { ws('workspace/ut-python-cpu') { init_git_win() unstash 'vc14_cpu' bat '''rmdir /s/q pkg_vc14_cpu - 7z x -y vc14_cpu.7z''' - bat """xcopy C:\\mxnet\\data data /E /I /Y - xcopy C:\\mxnet\\model model /E /I /Y - call activate py3 - set PYTHONPATH=${env.WORKSPACE}\\pkg_vc14_cpu\\python - del /S /Q ${env.WORKSPACE}\\pkg_vc14_cpu\\python\\*.pyc - C:\\mxnet\\test_cpu.bat""" - } + 7z x -y vc14_cpu.7z''' + bat """xcopy C:\\mxnet\\data data /E /I /Y + xcopy C:\\mxnet\\model model /E /I /Y + call activate py3 + set PYTHONPATH=${env.WORKSPACE}\\pkg_vc14_cpu\\python + del /S /Q ${env.WORKSPACE}\\pkg_vc14_cpu\\python\\*.pyc + C:\\mxnet\\test_cpu.bat""" + } } }, 'Python 2: GPU Win':{ - node('mxnetwindows') { + node('mxnetwindows-gpu') { ws('workspace/ut-python-gpu') { - init_git_win() - unstash 'vc14_gpu' - bat '''rmdir /s/q pkg_vc14_gpu - 7z x -y vc14_gpu.7z''' - bat """xcopy C:\\mxnet\\data data /E /I /Y - xcopy C:\\mxnet\\model model /E /I /Y - call activate py2 - set PYTHONPATH=${env.WORKSPACE}\\pkg_vc14_gpu\\python - del /S /Q ${env.WORKSPACE}\\pkg_vc14_gpu\\python\\*.pyc - C:\\mxnet\\test_gpu.bat""" + init_git_win() + unstash 'vc14_gpu' + bat '''rmdir /s/q pkg_vc14_gpu + 7z x -y vc14_gpu.7z''' + bat """xcopy C:\\mxnet\\data data /E /I /Y + xcopy C:\\mxnet\\model model /E /I /Y + call activate py2 + set PYTHONPATH=${env.WORKSPACE}\\pkg_vc14_gpu\\python + del /S /Q ${env.WORKSPACE}\\pkg_vc14_gpu\\python\\*.pyc + C:\\mxnet\\test_gpu.bat""" } } }, 'Python 3: GPU Win':{ - node('mxnetwindows') { + node('mxnetwindows-gpu') { ws('workspace/ut-python-gpu') { - init_git_win() - unstash 'vc14_gpu' - bat '''rmdir /s/q pkg_vc14_gpu - 7z x -y vc14_gpu.7z''' - bat """xcopy C:\\mxnet\\data data /E /I /Y - xcopy C:\\mxnet\\model model /E /I /Y - call activate py3 - set PYTHONPATH=${env.WORKSPACE}\\pkg_vc14_gpu\\python - del /S /Q ${env.WORKSPACE}\\pkg_vc14_gpu\\python\\*.pyc - C:\\mxnet\\test_gpu.bat""" + init_git_win() + unstash 'vc14_gpu' + bat '''rmdir /s/q pkg_vc14_gpu + 7z x -y vc14_gpu.7z''' + bat """xcopy C:\\mxnet\\data data /E /I /Y + xcopy C:\\mxnet\\model model /E /I /Y + call activate py3 + set PYTHONPATH=${env.WORKSPACE}\\pkg_vc14_gpu\\python + del /S /Q ${env.WORKSPACE}\\pkg_vc14_gpu\\python\\*.pyc + C:\\mxnet\\test_gpu.bat""" } } } } stage('Integration Test') { - parallel 'Python': { - node('mxnetlinux') { + parallel 'Python GPU': { + node('mxnetlinux-gpu') { ws('workspace/it-python-gpu') { init_git() unpack_lib('gpu') timeout(time: max_time, unit: 'MINUTES') { - sh "${docker_run} gpu PYTHONPATH=./python/ python example/image-classification/test_score.py" + sh "${docker_run} gpu --dockerbinary nvidia-docker PYTHONPATH=./python/ python example/image-classification/test_score.py" } } } }, - 'Caffe': { - node('mxnetlinux') { + 'Caffe GPU': { + node('mxnetlinux-gpu') { ws('workspace/it-caffe') { init_git() unpack_lib('gpu') timeout(time: max_time, unit: 'MINUTES') { - sh "${docker_run} caffe_gpu PYTHONPATH=/caffe/python:./python python tools/caffe_converter/test_converter.py" + sh "${docker_run} caffe_gpu --dockerbinary nvidia-docker PYTHONPATH=/caffe/python:./python python tools/caffe_converter/test_converter.py" } } } }, - 'cpp-package': { - node('mxnetlinux') { + 'cpp-package GPU': { + node('mxnetlinux-gpu') { ws('workspace/it-cpp-package') { init_git() unpack_lib('gpu') unstash 'cpp_test_score' timeout(time: max_time, unit: 'MINUTES') { - sh "${docker_run} gpu cpp-package/tests/ci_test.sh" + sh "${docker_run} gpu --dockerbinary nvidia-docker cpp-package/tests/ci_test.sh" } } } @@ -511,7 +530,7 @@ try { } stage('Deploy') { - node('mxnetlinux') { + node('mxnetlinux-cpu') { ws('workspace/docs') { if (env.BRANCH_NAME == "master") { init_git() @@ -524,13 +543,13 @@ try { // set build status to success at the end currentBuild.result = "SUCCESS" } catch (caughtError) { - node("mxnetlinux") { + node("mxnetlinux-cpu") { sh "echo caught ${caughtError}" err = caughtError currentBuild.result = "FAILURE" } } finally { - node("mxnetlinux") { + node("mxnetlinux-cpu") { // Only send email if master failed if (currentBuild.result == "FAILURE" && env.BRANCH_NAME == "master") { emailext body: 'Build for MXNet branch ${BRANCH_NAME} has broken. Please view the build at ${BUILD_URL}', replyTo: '${EMAIL}', subject: '[BUILD FAILED] Branch ${BRANCH_NAME} build ${BUILD_NUMBER}', to: '${EMAIL}' diff --git a/Makefile b/Makefile index 72dd26e..f506257 100644 --- a/Makefile +++ b/Makefile @@ -267,6 +267,7 @@ ifeq ($(CUDA_ARCH),) CUDA_ARCH += $(shell $(NVCC) -cuda $(COMPRESS) --x cu /dev/null -o /dev/null >/dev/null 2>&1 && \ echo $(COMPRESS)) endif +$(info Running CUDA_ARCH: $(CUDA_ARCH)) endif # ps-lite @@ -330,6 +331,9 @@ ifeq ($(USE_CUDA), 1) CFLAGS += -I$(ROOTDIR)/3rdparty/cub ALL_DEP += $(CUOBJ) $(EXTRA_CUOBJ) $(PLUGIN_CUOBJ) LDFLAGS += -lcuda -lcufft -lnvrtc + # Make sure to add stubs as fallback in order to be able to build + # without full CUDA install (especially if run without nvidia-docker) + LDFLAGS += -L/usr/local/cuda/lib64/stubs SCALA_PKG_PROFILE := $(SCALA_PKG_PROFILE)-gpu ifeq ($(USE_NCCL), 1) ifneq ($(USE_NCCL_PATH), NONE) diff --git a/perl-package/test.sh b/perl-package/test.sh index 1a4bd72..417e00a 100755 --- a/perl-package/test.sh +++ b/perl-package/test.sh @@ -29,4 +29,4 @@ make install || exit -1 cd ${MXNET_HOME}/perl-package/AI-MXNet/ perl Makefile.PL INSTALL_BASE=${MXNET_HOME}/perl5 -make test || exit -1 +make test TEST_VERBOSE=1 || exit -1 # Add debug output to test log diff --git a/tests/ci_build/Dockerfile.mklml_gpu b/tests/ci_build/Dockerfile.build_cuda similarity index 50% copy from tests/ci_build/Dockerfile.mklml_gpu copy to tests/ci_build/Dockerfile.build_cuda index 185681c..5fccec7 100644 --- a/tests/ci_build/Dockerfile.mklml_gpu +++ b/tests/ci_build/Dockerfile.build_cuda @@ -1,4 +1,6 @@ -FROM nvidia/cuda:7.5-cudnn5-devel +FROM nvidia/cuda:8.0-cudnn5-devel +# cuda8.0 has to be used because this is the first ubuntu16.04 container +# which is required due to OpenBLAS being incompatible with ubuntu14.04 # the reason we used a gpu base container because we are going to test MKLDNN # operator implementation against GPU implementation @@ -8,8 +10,17 @@ COPY install/ubuntu_install_python.sh /install/ RUN /install/ubuntu_install_python.sh COPY install/ubuntu_install_scala.sh /install/ RUN /install/ubuntu_install_scala.sh +COPY install/ubuntu_install_r.sh /install/ +RUN /install/ubuntu_install_r.sh +COPY install/ubuntu_install_perl.sh /install/ +RUN /install/ubuntu_install_perl.sh -RUN wget --no-check-certificate -O /tmp/mklml.tgz https://github.com/01org/mkl-dnn/releases/download/v0.10/mklml_lnx_2018.0.20170908.tgz +# Allows to run tasks on a CPU without nvidia-docker and GPU +COPY install/ubuntu_install_nvidia.sh /install/ +RUN /install/ubuntu_install_nvidia.sh + +# Add MKLML libraries +RUN wget --no-check-certificate -O /tmp/mklml.tgz https://github.com/01org/mkl-dnn/releases/download/v0.11/mklml_lnx_2018.0.1.20171007.tgz RUN tar -zxvf /tmp/mklml.tgz && cp -rf mklml_*/* /usr/local/ && rm -rf mklml_* ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib diff --git a/tests/ci_build/Dockerfile.caffe_gpu b/tests/ci_build/Dockerfile.caffe_gpu index 4f6522d..34c4625 100644 --- a/tests/ci_build/Dockerfile.caffe_gpu +++ b/tests/ci_build/Dockerfile.caffe_gpu @@ -1,4 +1,6 @@ -FROM nvidia/cuda:7.5-cudnn5-devel +FROM nvidia/cuda:8.0-cudnn5-devel +# cuda8.0 has to be used because this is the first ubuntu16.04 container +# which is required due to OpenBLAS being incompatible with ubuntu14.04 COPY install/ubuntu_install_core.sh /install/ RUN /install/ubuntu_install_core.sh @@ -18,6 +20,15 @@ RUN cd /; git clone http://github.com/BVLC/caffe.git; cd caffe; \ RUN echo "CPU_ONLY := 1" >> /caffe/Makefile.config +# Fixes https://github.com/BVLC/caffe/issues/5658 See https://github.com/intel/caffe/wiki/Ubuntu-16.04-or-15.10-Installation-Guide +RUN echo "INCLUDE_DIRS += /usr/lib /usr/lib/x86_64-linux-gnu /usr/include/hdf5/serial/ " >> /caffe/Makefile.config +RUN echo "LIBRARY_DIRS += /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial " >> /caffe/Makefile.config + +# Fixes https://github.com/BVLC/caffe/issues/4333 See https://github.com/intel/caffe/wiki/Ubuntu-16.04-or-15.10-Installation-Guide +# Note: This is only valid on Ubuntu16.04 - the version numbers are bound to the distribution +RUN ln -s /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.10.0.2 /usr/lib/x86_64-linux-gnu/libhdf5.so +RUN ln -s /usr/lib/x86_64-linux-gnu/libhdf5_serial_hl.so.10.0.2 /usr/lib/x86_64-linux-gnu/libhdf5_hl.so + RUN cd caffe; make all pycaffe -j$(nproc) RUN cd caffe/python; for req in $(cat requirements.txt); do pip2 install $req; done diff --git a/tests/ci_build/Dockerfile.cpu b/tests/ci_build/Dockerfile.cpu index c7bb0af..226054a 100644 --- a/tests/ci_build/Dockerfile.cpu +++ b/tests/ci_build/Dockerfile.cpu @@ -1,4 +1,4 @@ -FROM ubuntu:14.04 +FROM ubuntu:16.04 COPY install/ubuntu_install_core.sh /install/ RUN /install/ubuntu_install_core.sh diff --git a/tests/ci_build/Dockerfile.mklml_gpu b/tests/ci_build/Dockerfile.cpu_mklml similarity index 60% copy from tests/ci_build/Dockerfile.mklml_gpu copy to tests/ci_build/Dockerfile.cpu_mklml index 185681c..faa7864 100644 --- a/tests/ci_build/Dockerfile.mklml_gpu +++ b/tests/ci_build/Dockerfile.cpu_mklml @@ -1,6 +1,4 @@ -FROM nvidia/cuda:7.5-cudnn5-devel -# the reason we used a gpu base container because we are going to test MKLDNN -# operator implementation against GPU implementation +FROM ubuntu:16.04 COPY install/ubuntu_install_core.sh /install/ RUN /install/ubuntu_install_core.sh @@ -8,8 +6,13 @@ COPY install/ubuntu_install_python.sh /install/ RUN /install/ubuntu_install_python.sh COPY install/ubuntu_install_scala.sh /install/ RUN /install/ubuntu_install_scala.sh +COPY install/ubuntu_install_r.sh /install/ +RUN /install/ubuntu_install_r.sh +COPY install/ubuntu_install_perl.sh /install/ +RUN /install/ubuntu_install_perl.sh -RUN wget --no-check-certificate -O /tmp/mklml.tgz https://github.com/01org/mkl-dnn/releases/download/v0.10/mklml_lnx_2018.0.20170908.tgz +# Add MKLML library, compatiable with Ubuntu16.04 +RUN wget --no-check-certificate -O /tmp/mklml.tgz https://github.com/01org/mkl-dnn/releases/download/v0.11/mklml_lnx_2018.0.1.20171007.tgz RUN tar -zxvf /tmp/mklml.tgz && cp -rf mklml_*/* /usr/local/ && rm -rf mklml_* ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib diff --git a/tests/ci_build/Dockerfile.gpu b/tests/ci_build/Dockerfile.gpu index a2893a9..2483e62 100644 --- a/tests/ci_build/Dockerfile.gpu +++ b/tests/ci_build/Dockerfile.gpu @@ -1,4 +1,6 @@ -FROM nvidia/cuda:7.5-cudnn5-devel +FROM nvidia/cuda:8.0-cudnn5-devel +# cuda8.0 has to be used because this is the first ubuntu16.04 container +# which is required due to OpenBLAS being incompatible with ubuntu14.04 COPY install/ubuntu_install_core.sh /install/ RUN /install/ubuntu_install_core.sh diff --git a/tests/ci_build/Dockerfile.mklml_gpu b/tests/ci_build/Dockerfile.gpu_mklml similarity index 65% rename from tests/ci_build/Dockerfile.mklml_gpu rename to tests/ci_build/Dockerfile.gpu_mklml index 185681c..2c3564c 100644 --- a/tests/ci_build/Dockerfile.mklml_gpu +++ b/tests/ci_build/Dockerfile.gpu_mklml @@ -1,4 +1,6 @@ -FROM nvidia/cuda:7.5-cudnn5-devel +FROM nvidia/cuda:8.0-cudnn5-devel +# cuda8.0 has to be used because this is the first ubuntu16.04 container +# # which is required due to OpenBLAS being incompatible with ubuntu14.04 # the reason we used a gpu base container because we are going to test MKLDNN # operator implementation against GPU implementation @@ -9,7 +11,8 @@ RUN /install/ubuntu_install_python.sh COPY install/ubuntu_install_scala.sh /install/ RUN /install/ubuntu_install_scala.sh -RUN wget --no-check-certificate -O /tmp/mklml.tgz https://github.com/01org/mkl-dnn/releases/download/v0.10/mklml_lnx_2018.0.20170908.tgz +# Add MKLML library, compatible with Ubuntu16.04 +RUN wget --no-check-certificate -O /tmp/mklml.tgz https://github.com/01org/mkl-dnn/releases/download/v0.11/mklml_lnx_2018.0.1.20171007.tgz RUN tar -zxvf /tmp/mklml.tgz && cp -rf mklml_*/* /usr/local/ && rm -rf mklml_* ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib diff --git a/tests/ci_build/Dockerfile.lint b/tests/ci_build/Dockerfile.lint index b19b767..a72b3f8 100644 --- a/tests/ci_build/Dockerfile.lint +++ b/tests/ci_build/Dockerfile.lint @@ -1,5 +1,6 @@ # For lint test -FROM ubuntu:14.04 +FROM ubuntu:16.04 -RUN apt-get update && apt-get install -y python-pip +# Sudo is not present on ubuntu16.04 +RUN apt-get update && apt-get install -y python-pip sudo RUN pip install cpplint pylint diff --git a/tests/ci_build/ci_build.sh b/tests/ci_build/ci_build.sh index 79fcd86..512eb7a 100755 --- a/tests/ci_build/ci_build.sh +++ b/tests/ci_build/ci_build.sh @@ -55,6 +55,12 @@ if [[ "$1" == "-it" ]]; then shift 1 fi +if [[ "$1" == "--dockerbinary" ]]; then + DOCKER_BINARY="$2" + echo "Using custom Docker Engine: ${DOCKER_BINARY}" + shift 2 +fi + if [[ ! -f "${DOCKERFILE_PATH}" ]]; then echo "Invalid Dockerfile path: \"${DOCKERFILE_PATH}\"" exit 1 @@ -73,11 +79,15 @@ if [ "$#" -lt 1 ] || [ ! -e "${SCRIPT_DIR}/Dockerfile.${CONTAINER_TYPE}" ]; then exit 1 fi -# Use nvidia-docker if the container is GPU. -if [[ "${CONTAINER_TYPE}" == *"gpu"* ]]; then - DOCKER_BINARY="nvidia-docker" -else - DOCKER_BINARY="docker" +# Only set docker binary automatically if it has not been specified +if [[ -z "${DOCKER_BINARY}" ]]; then + # Use nvidia-docker if the container is GPU. + if [[ "${CONTAINER_TYPE}" == *"gpu"* ]]; then + DOCKER_BINARY="nvidia-docker" + else + DOCKER_BINARY="docker" + fi + echo "Automatically assuming ${DOCKER_BINARY} as docker binary" fi # Helper function to traverse directories up until given file is found. @@ -147,6 +157,7 @@ ${DOCKER_BINARY} run --rm --pid=host \ -e "CI_BUILD_UID=$(id -u)" \ -e "CI_BUILD_GROUP=$(id -g -n)" \ -e "CI_BUILD_GID=$(id -g)" \ + -e "CUDA_ARCH=-gencode arch=compute_52,code=[sm_52,compute_52] --fatbin-options -compress-all" \ ${CI_DOCKER_EXTRA_PARAMS[@]} \ ${DOCKER_IMG_NAME} \ ${PRE_COMMAND} \ diff --git a/tests/ci_build/install/ubuntu_install_core.sh b/tests/ci_build/install/ubuntu_install_core.sh index 4947574..eefd759 100755 --- a/tests/ci_build/install/ubuntu_install_core.sh +++ b/tests/ci_build/install/ubuntu_install_core.sh @@ -21,6 +21,9 @@ apt-get update && apt-get install -y \ build-essential git libopenblas-dev liblapack-dev libopencv-dev \ - libcurl4-openssl-dev libgtest-dev cmake wget unzip + libcurl4-openssl-dev libgtest-dev cmake wget unzip sudo + +# Link Openblas to Cblas as this link does not exist on ubuntu16.04 +ln -s /usr/lib/libopenblas.so /usr/lib/libcblas.so cd /usr/src/gtest && cmake CMakeLists.txt && make && cp *.a /usr/lib diff --git a/tests/ci_build/install/ubuntu_install_core.sh b/tests/ci_build/install/ubuntu_install_nvidia.sh similarity index 64% copy from tests/ci_build/install/ubuntu_install_core.sh copy to tests/ci_build/install/ubuntu_install_nvidia.sh index 4947574..71fde8e 100755 --- a/tests/ci_build/install/ubuntu_install_core.sh +++ b/tests/ci_build/install/ubuntu_install_nvidia.sh @@ -17,10 +17,15 @@ # specific language governing permissions and limitations # under the License. -# install libraries for building mxnet c++ core on ubuntu +# install nvidia libraries to compile and run CUDA without +# the necessity of nvidia-docker and a GPU -apt-get update && apt-get install -y \ - build-essential git libopenblas-dev liblapack-dev libopencv-dev \ - libcurl4-openssl-dev libgtest-dev cmake wget unzip +# Needed to run add-apt-repository +apt update && apt install -y software-properties-common -cd /usr/src/gtest && cmake CMakeLists.txt && make && cp *.a /usr/lib +add-apt-repository -y ppa:graphics-drivers + +# Retrieve ppa:graphics-drivers and install nvidia-drivers. +# Note: DEBIAN_FRONTEND required to skip the interactive setup steps +apt update && \ + DEBIAN_FRONTEND=noninteractive apt install -y nvidia-384 diff --git a/tests/ci_build/pip_tests/Dockerfile.in.pip_cpu b/tests/ci_build/pip_tests/Dockerfile.in.pip_cpu index dfd675b..de4629f 100644 --- a/tests/ci_build/pip_tests/Dockerfile.in.pip_cpu +++ b/tests/ci_build/pip_tests/Dockerfile.in.pip_cpu @@ -1,4 +1,4 @@ # -*- mode: dockerfile -*- # dockerfile to test pip installation on CPU -FROM ubuntu:14.04 +FROM ubuntu:16.04 diff --git a/tests/python/unittest/test_loss.py b/tests/python/unittest/test_loss.py index 8ee4bfa..e044df0 100644 --- a/tests/python/unittest/test_loss.py +++ b/tests/python/unittest/test_loss.py @@ -19,6 +19,7 @@ import mxnet as mx import numpy as np from mxnet import gluon from mxnet.test_utils import assert_almost_equal, default_context +import unittest def test_loss_ndarray(): @@ -160,6 +161,7 @@ def test_l1_loss(): assert mod.score(data_iter, eval_metric=mx.metric.Loss())[0][1] < 0.1 +@unittest.skip("flaky test. https://github.com/apache/incubator-mxnet/issues/8892") def test_ctc_loss(): loss = gluon.loss.CTCLoss() l = loss(mx.nd.ones((2,20,4)), mx.nd.array([[1,0,-1,-1],[2,1,1,-1]])) @@ -185,7 +187,7 @@ def test_ctc_loss(): l = loss(mx.nd.ones((2,25,4)), mx.nd.array([[2,1,3,3],[3,2,2,3]]), mx.nd.array([20,20]), mx.nd.array([2,3])) mx.test_utils.assert_almost_equal(l.asnumpy(), np.array([18.82820702, 16.50581741])) - +@unittest.skip("flaky test. https://github.com/apache/incubator-mxnet/issues/8892") def test_ctc_loss_train(): np.random.seed(1234) N = 20 -- To stop receiving notification emails like this one, please contact ['"comm...@mxnet.apache.org" <comm...@mxnet.apache.org>'].