This is an automated email from the ASF dual-hosted git repository.
jevans pushed a commit to branch v1.9.x
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/v1.9.x by this push:
new 67abb85131 [v1.9.x] Refactor dockerfiles in CI, migrate some ubuntu
docker containers to use docker-compose. Update CI to use Cuda 11.7 (#21126)
67abb85131 is described below
commit 67abb851316349aaa035a7a97220476e98c9b8b4
Author: Joe Evans <[email protected]>
AuthorDate: Mon Aug 22 13:58:58 2022 -0700
[v1.9.x] Refactor dockerfiles in CI, migrate some ubuntu docker containers
to use docker-compose. Update CI to use Cuda 11.7 (#21126)
* Remove deprecated dockerfiles.
* Update documentation to use different image.
* Install Scala in centos7 CD container and build tools.
* Update static scala build to use CD container, change julia container.
* Removed deprecated Jenkins pipeline files, remove old disabled build
steps.
* Add new base Dockerfile for docker-compose.
* Migrate ubuntu cuda containers to docker-compose.
* Build python from source on ubuntu for portability.
* Remove old dockerfiles, upgrade nightly gpu image to cuda 11.7.
* Remove Cuda versions from runtime function names to simplify.
* Update Jenkins pipelines to use newer Cuda containers.
* Install LLVM before TVM.
* Fix ubuntu TVM install script (was failing but returning true.)
* Move cmake install into unified script.
* Move cmake install for ubuntu into centralized script.
* Update cudnn version passed to builds.
* Consolidate installation of packages for efficiency.
* Remove unused containers from docker-compose config.
* Fix pylint.
* Set LD_LIBRARY_PATH on ubuntu_gpu images to find libcuda.so.
* Set CUB_IGNORE_DEPRECATED_CPP_DIALECT to prevent build failures with
gcc-4.8 + Cuda 11.7.
* Install sqlite headers/library before building python on ubuntu.
* Revert "Remove unused containers from docker-compose config."
This reverts commit 5de82dfe8f092f589fe18c010ac42b07faf21985.
* Revert "Set CUB_IGNORE_DEPRECATED_CPP_DIALECT to prevent build failures
with gcc-4.8 + Cuda 11.7."
This reverts commit e6496607c0673965c2153ae7167e1785eac26bfb.
* Allow building CUB with c++11 to prevent failures on newer cuda versions.
* Set variable only on gpu make builds.
* Use docker-compose to also build ubuntu_cpu image.
* We no longer need to enable python3.8 on aarch64 since we are building
from source now.
* Add Cuda 11.1 and 11.3 centos7 images which is used by CD testing phase.
* Don't install python-opencv, we are installing the module via pip instead.
* Change Makefile to set CUB_IGNORE_DEPRECATED_CPP_DIALECT when using Cuda,
not only for < 11.0.
* Don't pin down h5py (old versions do not work on aarch64.)
* Conditionally install different versions of h5py dependending on
architecture.
* Fix value for platform_machine.
* Don't install h5py on aarch64 at all.
* Set USE_LAPATH_PATH to correct path on ubuntu 18.04.
* Rearrange dockerfiles to build more efficiently when small changes occur.
Split python install into 2 steps: building python and install requirements.
* Since we are not using multi-stage builds, do not specify target to
ensure docker cache works as expected.
* When building docker-compose based containers, pull the latest version
for caching before building.
* When pulling docker-compose images, pass quiet option to squell CI logs.
* When pulling docker-compose images, pass quiet option to squell CI logs.
* Clean up docker cache build code.
---
Makefile | 3 +-
ci/build.py | 18 ++--
ci/dev_menu.py | 2 +-
ci/docker/Dockerfile.build.centos7 | 15 ++-
ci/docker/Dockerfile.build.centos7_aarch64_cpu | 11 +-
ci/docker/Dockerfile.build.centos7_cd | 24 +++--
ci/docker/Dockerfile.build.ubuntu_base_cpu | 40 -------
ci/docker/Dockerfile.build.ubuntu_base_gpu | 43 --------
ci/docker/Dockerfile.build.ubuntu_build_cuda | 60 -----------
ci/docker/Dockerfile.build.ubuntu_build_cuda110 | 48 ---------
ci/docker/Dockerfile.build.ubuntu_cpu | 12 ++-
ci/docker/Dockerfile.build.ubuntu_cpu_c | 5 +-
ci/docker/Dockerfile.build.ubuntu_cpu_julia | 77 --------------
ci/docker/Dockerfile.build.ubuntu_cpu_lite | 5 +-
ci/docker/Dockerfile.build.ubuntu_cpu_python | 8 +-
ci/docker/Dockerfile.build.ubuntu_cpu_r | 3 +
ci/docker/Dockerfile.build.ubuntu_cpu_scala | 14 ++-
...buntu_gpu_cu102 => Dockerfile.build.ubuntu_gpu} | 25 +++--
ci/docker/Dockerfile.build.ubuntu_gpu_cu100 | 75 -------------
ci/docker/Dockerfile.build.ubuntu_gpu_cu101 | 75 -------------
ci/docker/Dockerfile.build.ubuntu_gpu_cu110 | 43 --------
ci/docker/Dockerfile.build.ubuntu_gpu_cu112 | 43 --------
ci/docker/Dockerfile.build.ubuntu_gpu_tensorrt | 9 +-
ci/docker/Dockerfile.build.ubuntu_nightly_cpu | 12 ++-
ci/docker/Dockerfile.build.ubuntu_nightly_gpu | 25 +++--
ci/docker/Dockerfile.publish.test.centos7_cpu | 38 -------
ci/docker/Dockerfile.publish.test.centos7_gpu | 38 -------
ci/docker/Dockerfile.publish.test.ubuntu1404_cpu | 39 -------
ci/docker/Dockerfile.publish.test.ubuntu1404_gpu | 40 -------
ci/docker/Dockerfile.publish.test.ubuntu1604_cpu | 39 -------
ci/docker/Dockerfile.publish.test.ubuntu1604_gpu | 39 -------
ci/docker/Dockerfile.publish.test.ubuntu1804_cpu | 41 --------
ci/docker/Dockerfile.publish.test.ubuntu1804_gpu | 41 --------
ci/docker/Dockerfile.publish.ubuntu1404_cpu | 39 -------
ci/docker/Dockerfile.publish.ubuntu1404_gpu | 39 -------
ci/docker/Dockerfile.publish.ubuntu1604_cpu | 44 --------
ci/docker/Dockerfile.publish.ubuntu1604_gpu | 44 --------
ci/docker/docker-compose.yml | 117 +++++++++++++++++++--
ci/docker/install/centos7_base.sh | 8 +-
ci/docker/install/centos7_core.sh | 46 ++++----
ci/docker/install/centos7_python.sh | 3 -
ci/docker/install/requirements | 2 +-
ci/docker/install/requirements_aarch64 | 34 ------
ci/docker/install/ubuntu_core.sh | 8 --
ci/docker/install/ubuntu_publish.sh | 96 -----------------
ci/docker/install/ubuntu_python.sh | 25 ++---
ci/docker/install/ubuntu_tutorials.sh | 2 +-
ci/docker/install/ubuntu_tvm.sh | 9 +-
ci/docker/runtime_functions.sh | 40 ++-----
ci/docker_cache.py | 7 +-
ci/jenkins/Jenkins_steps.groovy | 96 +++++------------
ci/jenkins/Jenkinsfile_unix_gpu_cu110 | 52 ---------
ci/publish/Jenkinsfile | 110 -------------------
python/mxnet/contrib/text/utils.py | 2 +-
python/mxnet/numpy/multiarray.py | 4 +-
python/mxnet/optimizer/optimizer.py | 4 +-
tests/nightly/Jenkinsfile | 28 -----
tests/nightly/JenkinsfileForBinaries | 4 +-
58 files changed, 316 insertions(+), 1557 deletions(-)
diff --git a/Makefile b/Makefile
index d4e221d4bb..d03505f7bf 100644
--- a/Makefile
+++ b/Makefile
@@ -579,8 +579,9 @@ ALL_DEP = $(OBJ) $(EXTRA_OBJ) $(PLUGIN_OBJ) $(LIB_DEP)
ifeq ($(USE_CUDA), 1)
CUDA_VERSION_MAJOR := $(shell $(NVCC) --version | grep "release" | awk
'{print $$6}' | cut -c2- | cut -d '.' -f1)
+ CFLAGS += -DCUB_IGNORE_DEPRECATED_CPP_DIALECT
ifeq ($(shell test $(CUDA_VERSION_MAJOR) -lt 11; echo $$?), 0)
- CFLAGS += -I$(ROOTDIR)/3rdparty/nvidia_cub
-DCUB_IGNORE_DEPRECATED_CPP_DIALECT
+ CFLAGS += -I$(ROOTDIR)/3rdparty/nvidia_cub
endif
ALL_DEP += $(CUOBJ) $(EXTRA_CUOBJ) $(PLUGIN_CUOBJ)
diff --git a/ci/build.py b/ci/build.py
index 1998f58ee2..c3de6c9271 100755
--- a/ci/build.py
+++ b/ci/build.py
@@ -187,11 +187,11 @@ def build_docker(platform: str, registry: str,
num_retries: int, no_cache: bool,
env["DOCKER_CACHE_REGISTRY"] = registry
@retry(subprocess.CalledProcessError, tries=num_retries)
- def run_cmd(env=None):
- logging.info("Running command: '%s'", ' '.join(cmd))
- check_call(cmd, env=env)
+ def run_cmd(c, e):
+ logging.info("Running command: '%s'", ' '.join(c))
+ check_call(c, env=e)
- run_cmd(env=env)
+ run_cmd(cmd, env)
# Get image id by reading the tag. It's guaranteed (except race condition)
that the tag exists. Otherwise, the
# check_call would have failed
@@ -308,23 +308,21 @@ def list_platforms(arch=machine()) -> str:
def load_docker_cache(platform, tag, docker_registry) -> None:
"""Imports tagged container from the given docker registry"""
if docker_registry:
+ env = os.environ.copy()
+ env["DOCKER_CACHE_REGISTRY"] = docker_registry
if is_docker_compose(platform):
docker_compose_platform = platform.split(".")[1] if any(x in
platform for x in ['build.', 'publish.']) else platform
- env = os.environ.copy()
- env["DOCKER_CACHE_REGISTRY"] = docker_registry
if "dkr.ecr" in docker_registry:
try:
import docker_cache
docker_cache._ecr_login(docker_registry)
except Exception:
logging.exception('Unable to login to ECR...')
- cmd = ['docker-compose', '-f', 'docker/docker-compose.yml',
'pull', docker_compose_platform]
- logging.info("Running command: 'DOCKER_CACHE_REGISTRY=%s %s'",
docker_registry, ' '.join(cmd))
+ cmd = ['docker-compose', '-f', 'docker/docker-compose.yml',
'pull', '--quiet', docker_compose_platform]
+ logging.info("Running command: '%s'", ' '.join(cmd))
check_call(cmd, env=env)
return
- env = os.environ.copy()
- env["DOCKER_CACHE_REGISTRY"] = docker_registry
# noinspection PyBroadException
try:
import docker_cache
diff --git a/ci/dev_menu.py b/ci/dev_menu.py
index 962e4ecfe0..863907b8ac 100755
--- a/ci/dev_menu.py
+++ b/ci/dev_menu.py
@@ -134,7 +134,7 @@ COMMANDS = OrderedDict([
('[Docker] Build the Java API docs - outputs to
"docs/scala-package/build/docs/java"',
"ci/build.py --platform ubuntu_cpu_scala /work/runtime_functions.sh
build_java_docs"),
('[Docker] Build the Julia API docs - outputs to "julia/docs/site/"',
- "ci/build.py --platform ubuntu_cpu_julia /work/runtime_functions.sh
build_julia_docs"),
+ "ci/build.py --platform ubuntu_cpu /work/runtime_functions.sh
build_julia_docs"),
('[Docker] Build the R API docs - outputs to
"R-package/build/mxnet-r-reference-manual.pdf"',
"ci/build.py --platform ubuntu_cpu_r /work/runtime_functions.sh
build_r_docs"),
('[Docker] Build the Scala API docs - outputs to
"scala-package/docs/build/docs/scala"',
diff --git a/ci/docker/Dockerfile.build.centos7
b/ci/docker/Dockerfile.build.centos7
index 8c0df1d91f..2d285dd8ef 100644
--- a/ci/docker/Dockerfile.build.centos7
+++ b/ci/docker/Dockerfile.build.centos7
@@ -31,20 +31,27 @@
# "--target" option or docker-compose.yml
####################################################################################################
ARG BASE_IMAGE
-FROM $BASE_IMAGE AS base
+FROM $BASE_IMAGE
WORKDIR /work/deps
COPY install/centos7_core.sh /work/
RUN /work/centos7_core.sh
+
+COPY install/centos7_cmake.sh /work/
+RUN /work/centos7_cmake.sh
+
COPY install/centos7_ccache.sh /work/
RUN /work/centos7_ccache.sh
-COPY install/centos7_python.sh /work/
-COPY install/requirements /work/
-RUN /work/centos7_python.sh
+
COPY install/centos7_scala.sh /work/
RUN /work/centos7_scala.sh
+COPY install/centos7_python.sh /work/
+RUN /work/centos7_python.sh
+COPY install/requirements /work/
+RUN pip3 install -r /work/requirements
+
ARG USER_ID=0
COPY install/centos7_adduser.sh /work/
RUN /work/centos7_adduser.sh
diff --git a/ci/docker/Dockerfile.build.centos7_aarch64_cpu
b/ci/docker/Dockerfile.build.centos7_aarch64_cpu
index 46c3e253dd..c4e98dc2ad 100644
--- a/ci/docker/Dockerfile.build.centos7_aarch64_cpu
+++ b/ci/docker/Dockerfile.build.centos7_aarch64_cpu
@@ -55,11 +55,6 @@ SHELL [ "/usr/bin/scl", "enable", "devtoolset-10" ]
# Fix the en_DK.UTF-8 locale to test locale invariance
RUN localedef -i en_DK -f UTF-8 en_DK.UTF-8
-# Install Python and dependency packages
-COPY install/centos7_python.sh /work/
-COPY install/requirements /work/
-RUN /work/centos7_python.sh
-
COPY install/centos7_cmake.sh /work/
RUN /work/centos7_cmake.sh
@@ -71,6 +66,12 @@ RUN cd /usr/local/src && \
rm -rf arm-performance-libraries_21.0_RHEL-7_gcc-8.2.tar
arm-performance-libraries_21.0_RHEL-7_gcc-8.2
ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/opt/arm/armpl_21.0_gcc-8.2/lib
+# Install Python and dependency packages
+COPY install/centos7_python.sh /work/
+RUN /work/centos7_python.sh
+COPY install/requirements /work/
+RUN pip3 install -r /work/requirements
+
ARG USER_ID=0
COPY install/centos7_adduser.sh /work/
RUN /work/centos7_adduser.sh
diff --git a/ci/docker/Dockerfile.build.centos7_cd
b/ci/docker/Dockerfile.build.centos7_cd
index 02bd695e7e..04f6d0e2da 100644
--- a/ci/docker/Dockerfile.build.centos7_cd
+++ b/ci/docker/Dockerfile.build.centos7_cd
@@ -25,23 +25,27 @@ WORKDIR /work/deps
COPY install/centos7_base.sh /work/
RUN /work/centos7_base.sh
-COPY install/centos7_python.sh /work/
-COPY install/requirements /work/
-RUN /work/centos7_python.sh
+COPY install/centos7_scala.sh /work/
+RUN /work/centos7_scala.sh
-ARG USER_ID=0
-COPY install/centos7_adduser.sh /work/
-RUN /work/centos7_adduser.sh
+# Install cmake
+COPY install/centos7_cmake.sh /work/
+RUN /work/centos7_cmake.sh
# Install tools for static dependency builds
-RUN yum install -y sudo patchelf nasm automake libtool file gcc-c++ gcc
gcc-gfortran
+RUN yum install -y sudo patchelf nasm automake libtool file gcc-c++ gcc
gcc-gfortran which
# Allow jenkins user to use sudo for installing cuda libraries
RUN echo "jenkins_slave ALL=(root) NOPASSWD: /usr/bin/yum,
/usr/bin/yum-config-manager" >> /etc/sudoers.d/10_jenkins_slave
-# Install cmake
-COPY install/centos7_cmake.sh /work/
-RUN /work/centos7_cmake.sh
+COPY install/centos7_python.sh /work/
+RUN /work/centos7_python.sh
+COPY install/requirements /work/
+RUN pip3 install -r /work/requirements
+
+ARG USER_ID=0
+COPY install/centos7_adduser.sh /work/
+RUN /work/centos7_adduser.sh
ENV PYTHONPATH=./python/
WORKDIR /work/mxnet
diff --git a/ci/docker/Dockerfile.build.ubuntu_base_cpu
b/ci/docker/Dockerfile.build.ubuntu_base_cpu
deleted file mode 100644
index a75ed0255d..0000000000
--- a/ci/docker/Dockerfile.build.ubuntu_base_cpu
+++ /dev/null
@@ -1,40 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to run the MXNet Installation Tests on Ubuntu 16.04
-# This should run in an empty docker with ubuntu and cuda.
-
-FROM ubuntu:16.04
-
-WORKDIR /work/deps
-
-RUN apt-get update && apt-get -y install sudo
-
-# Always last
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY install/ubuntu_runas_sudo.sh /work/
-RUN /work/ubuntu_runas_sudo.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
-ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
diff --git a/ci/docker/Dockerfile.build.ubuntu_base_gpu
b/ci/docker/Dockerfile.build.ubuntu_base_gpu
deleted file mode 100644
index 94e6437e57..0000000000
--- a/ci/docker/Dockerfile.build.ubuntu_base_gpu
+++ /dev/null
@@ -1,43 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to run the MXNet Installation Tests on Ubuntu 16.04
-# This should run in an empty docker with ubuntu and cuda.
-
-FROM nvidia/cuda:10.1-devel-ubuntu16.04
-
-WORKDIR /work/deps
-
-RUN apt-get update && apt-get -y install sudo
-
-ENV CUDNN_VERSION=7.6.0.64
-COPY install/ubuntu_cudnn.sh /work/
-RUN /work/ubuntu_cudnn.sh
-
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY install/ubuntu_runas_sudo.sh /work/
-RUN /work/ubuntu_runas_sudo.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
-ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
diff --git a/ci/docker/Dockerfile.build.ubuntu_build_cuda
b/ci/docker/Dockerfile.build.ubuntu_build_cuda
deleted file mode 100644
index f5e11100e5..0000000000
--- a/ci/docker/Dockerfile.build.ubuntu_build_cuda
+++ /dev/null
@@ -1,60 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to build MXNet on Ubuntu 18.04 for GPU but on
-# a CPU-only instance. This restriction is caused by the CPP-
-# package generation, requiring the actual CUDA library to be
-# present
-
-FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04
-
-WORKDIR /work/deps
-
-COPY install/ubuntu_core.sh /work/
-RUN /work/ubuntu_core.sh
-COPY install/deb_ubuntu_ccache.sh /work/
-RUN /work/deb_ubuntu_ccache.sh
-COPY install/ubuntu_python.sh /work/
-COPY install/requirements /work/
-RUN /work/ubuntu_python.sh
-COPY install/ubuntu_scala.sh /work/
-COPY install/sbt.gpg /work/
-RUN /work/ubuntu_scala.sh
-COPY install/ubuntu_r.sh /work/
-RUN /work/ubuntu_r.sh
-COPY install/ubuntu_perl.sh /work/
-RUN /work/ubuntu_perl.sh
-COPY install/ubuntu_clang.sh /work/
-RUN /work/ubuntu_clang.sh
-COPY install/ubuntu_binutils.sh /work/
-RUN /work/ubuntu_binutils.sh
-
-# Special case because the CPP-Package requires the CUDA runtime libs
-# and not only stubs (which are provided by the base image)
-COPY install/ubuntu_nvidia.sh /work/
-RUN /work/ubuntu_nvidia.sh
-
-# Keep this at the end since this command is not cachable
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
diff --git a/ci/docker/Dockerfile.build.ubuntu_build_cuda110
b/ci/docker/Dockerfile.build.ubuntu_build_cuda110
deleted file mode 100644
index 3dcb3d6595..0000000000
--- a/ci/docker/Dockerfile.build.ubuntu_build_cuda110
+++ /dev/null
@@ -1,48 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to build MXNet on Ubuntu 16.04 for GPU but on
-# a CPU-only instance. This restriction is caused by the CPP-
-# package generation, requiring the actual CUDA library to be
-# present
-
-FROM nvidia/cuda:11.0-cudnn8-devel-ubuntu16.04
-
-WORKDIR /work/deps
-
-COPY install/ubuntu_core.sh /work/
-RUN /work/ubuntu_core.sh
-COPY install/deb_ubuntu_ccache.sh /work/
-RUN /work/deb_ubuntu_ccache.sh
-COPY install/ubuntu_python.sh /work/
-COPY install/requirements /work/
-RUN /work/ubuntu_python.sh
-COPY install/ubuntu_binutils.sh /work/
-RUN /work/ubuntu_binutils.sh
-
-# Keep this at the end since this command is not cachable
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
-ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda/compat
-
diff --git a/ci/docker/Dockerfile.build.ubuntu_cpu
b/ci/docker/Dockerfile.build.ubuntu_cpu
index c8f23edc11..7c222edbb3 100644
--- a/ci/docker/Dockerfile.build.ubuntu_cpu
+++ b/ci/docker/Dockerfile.build.ubuntu_cpu
@@ -25,13 +25,12 @@ WORKDIR /work/deps
COPY install/ubuntu_core.sh /work/
RUN /work/ubuntu_core.sh
+COPY install/ubuntu_cmake.sh /work/
+RUN /work/ubuntu_cmake.sh
+
COPY install/deb_ubuntu_ccache.sh /work/
RUN /work/deb_ubuntu_ccache.sh
-COPY install/ubuntu_python.sh /work/
-COPY install/requirements /work/
-RUN /work/ubuntu_python.sh
-
COPY install/ubuntu_scala.sh /work/
COPY install/sbt.gpg /work/
RUN /work/ubuntu_scala.sh
@@ -60,6 +59,11 @@ RUN /work/ubuntu_mkl.sh
COPY install/ubuntu_caffe.sh /work/
RUN /work/ubuntu_caffe.sh
+COPY install/ubuntu_python.sh /work/
+RUN /work/ubuntu_python.sh
+COPY install/requirements /work/
+RUN pip3 install -r /work/requirements
+
COPY install/ubuntu_onnx.sh /work/
RUN /work/ubuntu_onnx.sh
diff --git a/ci/docker/Dockerfile.build.ubuntu_cpu_c
b/ci/docker/Dockerfile.build.ubuntu_cpu_c
index afe9eea260..42ab7c8f27 100644
--- a/ci/docker/Dockerfile.build.ubuntu_cpu_c
+++ b/ci/docker/Dockerfile.build.ubuntu_cpu_c
@@ -25,6 +25,9 @@ WORKDIR /work/deps
COPY install/ubuntu_core.sh /work/
RUN /work/ubuntu_core.sh
+COPY install/ubuntu_cmake.sh /work/
+RUN /work/ubuntu_cmake.sh
+
COPY install/deb_ubuntu_ccache.sh /work/
RUN /work/deb_ubuntu_ccache.sh
@@ -32,4 +35,4 @@ RUN apt-get update && apt-get install -y doxygen graphviz
COPY runtime_functions.sh /work/
-WORKDIR /work/mxnet
\ No newline at end of file
+WORKDIR /work/mxnet
diff --git a/ci/docker/Dockerfile.build.ubuntu_cpu_julia
b/ci/docker/Dockerfile.build.ubuntu_cpu_julia
deleted file mode 100644
index c8f23edc11..0000000000
--- a/ci/docker/Dockerfile.build.ubuntu_cpu_julia
+++ /dev/null
@@ -1,77 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to build and run MXNet on Ubuntu 18.04 for CPU
-
-FROM ubuntu:18.04
-
-WORKDIR /work/deps
-
-COPY install/ubuntu_core.sh /work/
-RUN /work/ubuntu_core.sh
-
-COPY install/deb_ubuntu_ccache.sh /work/
-RUN /work/deb_ubuntu_ccache.sh
-
-COPY install/ubuntu_python.sh /work/
-COPY install/requirements /work/
-RUN /work/ubuntu_python.sh
-
-COPY install/ubuntu_scala.sh /work/
-COPY install/sbt.gpg /work/
-RUN /work/ubuntu_scala.sh
-
-COPY install/ubuntu_clojure.sh /work/
-RUN /work/ubuntu_clojure.sh
-
-COPY install/ubuntu_r.sh /work/
-RUN /work/ubuntu_r.sh
-
-COPY install/ubuntu_perl.sh /work/
-RUN /work/ubuntu_perl.sh
-
-COPY install/ubuntu_julia.sh /work/
-RUN /work/ubuntu_julia.sh
-
-COPY install/ubuntu_clang.sh /work/
-RUN /work/ubuntu_clang.sh
-
-COPY install/ubuntu_gcc8.sh /work/
-RUN /work/ubuntu_gcc8.sh
-
-COPY install/ubuntu_mkl.sh /work/
-RUN /work/ubuntu_mkl.sh
-
-COPY install/ubuntu_caffe.sh /work/
-RUN /work/ubuntu_caffe.sh
-
-COPY install/ubuntu_onnx.sh /work/
-RUN /work/ubuntu_onnx.sh
-
-COPY install/ubuntu_docs.sh /work/
-RUN /work/ubuntu_docs.sh
-
-# Always last
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
diff --git a/ci/docker/Dockerfile.build.ubuntu_cpu_lite
b/ci/docker/Dockerfile.build.ubuntu_cpu_lite
index ff97e68da9..9a2080f440 100644
--- a/ci/docker/Dockerfile.build.ubuntu_cpu_lite
+++ b/ci/docker/Dockerfile.build.ubuntu_cpu_lite
@@ -25,6 +25,9 @@ WORKDIR /work/deps
COPY install/ubuntu_core.sh /work/
RUN /work/ubuntu_core.sh
+COPY install/ubuntu_cmake.sh /work/
+RUN /work/ubuntu_cmake.sh
+
COPY install/deb_ubuntu_ccache.sh /work/
RUN /work/deb_ubuntu_ccache.sh
@@ -42,4 +45,4 @@ RUN /work/ubuntu_adduser.sh
COPY runtime_functions.sh /work/
-WORKDIR /work/mxnet
\ No newline at end of file
+WORKDIR /work/mxnet
diff --git a/ci/docker/Dockerfile.build.ubuntu_cpu_python
b/ci/docker/Dockerfile.build.ubuntu_cpu_python
index 74eae602f7..e1990fd6dd 100644
--- a/ci/docker/Dockerfile.build.ubuntu_cpu_python
+++ b/ci/docker/Dockerfile.build.ubuntu_cpu_python
@@ -25,12 +25,16 @@ WORKDIR /work/deps
COPY install/ubuntu_core.sh /work/
RUN /work/ubuntu_core.sh
+COPY install/ubuntu_cmake.sh /work/
+RUN /work/ubuntu_cmake.sh
+
COPY install/deb_ubuntu_ccache.sh /work/
RUN /work/deb_ubuntu_ccache.sh
COPY install/ubuntu_python.sh /work/
-COPY install/requirements /work/
RUN /work/ubuntu_python.sh
+COPY install/requirements /work/
+RUN pip3 install -r /work/requirements
COPY install/ubuntu_onnx.sh /work/
RUN /work/ubuntu_onnx.sh
@@ -46,4 +50,4 @@ RUN /work/ubuntu_adduser.sh
COPY runtime_functions.sh /work/
-WORKDIR /work/mxnet
\ No newline at end of file
+WORKDIR /work/mxnet
diff --git a/ci/docker/Dockerfile.build.ubuntu_cpu_r
b/ci/docker/Dockerfile.build.ubuntu_cpu_r
index 3d17e09d87..e59cb0993f 100644
--- a/ci/docker/Dockerfile.build.ubuntu_cpu_r
+++ b/ci/docker/Dockerfile.build.ubuntu_cpu_r
@@ -25,6 +25,9 @@ WORKDIR /work/deps
COPY install/ubuntu_core.sh /work/
RUN /work/ubuntu_core.sh
+COPY install/ubuntu_cmake.sh /work/
+RUN /work/ubuntu_cmake.sh
+
COPY install/deb_ubuntu_ccache.sh /work/
RUN /work/deb_ubuntu_ccache.sh
diff --git a/ci/docker/Dockerfile.build.ubuntu_cpu_scala
b/ci/docker/Dockerfile.build.ubuntu_cpu_scala
index 4a5ffe2784..d3c8f42014 100644
--- a/ci/docker/Dockerfile.build.ubuntu_cpu_scala
+++ b/ci/docker/Dockerfile.build.ubuntu_cpu_scala
@@ -25,13 +25,12 @@ WORKDIR /work/deps
COPY install/ubuntu_core.sh /work/
RUN /work/ubuntu_core.sh
+COPY install/ubuntu_cmake.sh /work/
+RUN /work/ubuntu_cmake.sh
+
COPY install/deb_ubuntu_ccache.sh /work/
RUN /work/deb_ubuntu_ccache.sh
-COPY install/ubuntu_python.sh /work/
-COPY install/requirements /work/
-RUN /work/ubuntu_python.sh
-
COPY install/ubuntu_scala.sh /work/
COPY install/sbt.gpg /work/
RUN /work/ubuntu_scala.sh
@@ -39,6 +38,11 @@ RUN /work/ubuntu_scala.sh
COPY install/ubuntu_clojure.sh /work/
RUN /work/ubuntu_clojure.sh
+COPY install/ubuntu_python.sh /work/
+RUN /work/ubuntu_python.sh
+COPY install/requirements /work/
+RUN pip3 install -r /work/requirements
+
# Always last
ARG USER_ID=0
ARG GROUP_ID=0
@@ -47,4 +51,4 @@ RUN /work/ubuntu_adduser.sh
COPY runtime_functions.sh /work/
-WORKDIR /work/mxnet
\ No newline at end of file
+WORKDIR /work/mxnet
diff --git a/ci/docker/Dockerfile.build.ubuntu_gpu_cu102
b/ci/docker/Dockerfile.build.ubuntu_gpu
similarity index 88%
rename from ci/docker/Dockerfile.build.ubuntu_gpu_cu102
rename to ci/docker/Dockerfile.build.ubuntu_gpu
index 21dd41fdbd..e8fd499d92 100644
--- a/ci/docker/Dockerfile.build.ubuntu_gpu_cu102
+++ b/ci/docker/Dockerfile.build.ubuntu_gpu
@@ -16,22 +16,22 @@
# specific language governing permissions and limitations
# under the License.
#
-# Dockerfile to run MXNet on Ubuntu 18.04 for GPU
+# Dockerfile base to run MXNet on Ubuntu 18.04 for GPU
-FROM nvidia/cuda:10.2-cudnn8-devel-ubuntu18.04
+ARG BASE_IMAGE
+FROM $BASE_IMAGE
WORKDIR /work/deps
COPY install/ubuntu_core.sh /work/
RUN /work/ubuntu_core.sh
+COPY install/ubuntu_cmake.sh /work/
+RUN /work/ubuntu_cmake.sh
+
COPY install/deb_ubuntu_ccache.sh /work/
RUN /work/deb_ubuntu_ccache.sh
-COPY install/ubuntu_python.sh /work/
-COPY install/requirements /work/
-RUN /work/ubuntu_python.sh
-
COPY install/ubuntu_scala.sh /work/
COPY install/sbt.gpg /work/
RUN /work/ubuntu_scala.sh
@@ -45,15 +45,20 @@ RUN /work/ubuntu_perl.sh
COPY install/ubuntu_clang.sh /work/
RUN /work/ubuntu_clang.sh
-COPY install/ubuntu_tvm.sh /work/
-RUN /work/ubuntu_tvm.sh
-
COPY install/ubuntu_llvm.sh /work/
RUN /work/ubuntu_llvm.sh
COPY install/ubuntu_caffe.sh /work/
RUN /work/ubuntu_caffe.sh
+COPY install/ubuntu_python.sh /work/
+RUN /work/ubuntu_python.sh
+COPY install/requirements /work/
+RUN pip3 install -r /work/requirements
+
+COPY install/ubuntu_tvm.sh /work/
+RUN /work/ubuntu_tvm.sh
+
COPY install/ubuntu_onnx.sh /work/
RUN /work/ubuntu_onnx.sh
@@ -72,4 +77,6 @@ RUN /work/ubuntu_adduser.sh
COPY runtime_functions.sh /work/
+ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda/compat
+
WORKDIR /work/mxnet
diff --git a/ci/docker/Dockerfile.build.ubuntu_gpu_cu100
b/ci/docker/Dockerfile.build.ubuntu_gpu_cu100
deleted file mode 100644
index b48dfa598e..0000000000
--- a/ci/docker/Dockerfile.build.ubuntu_gpu_cu100
+++ /dev/null
@@ -1,75 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to run MXNet on Ubuntu 18.04 for GPU
-
-FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04
-
-WORKDIR /work/deps
-
-COPY install/ubuntu_core.sh /work/
-RUN /work/ubuntu_core.sh
-
-COPY install/deb_ubuntu_ccache.sh /work/
-RUN /work/deb_ubuntu_ccache.sh
-
-COPY install/ubuntu_python.sh /work/
-COPY install/requirements /work/
-RUN /work/ubuntu_python.sh
-
-COPY install/ubuntu_scala.sh /work/
-COPY install/sbt.gpg /work/
-RUN /work/ubuntu_scala.sh
-
-COPY install/ubuntu_r.sh /work/
-RUN /work/ubuntu_r.sh
-
-COPY install/ubuntu_perl.sh /work/
-RUN /work/ubuntu_perl.sh
-
-COPY install/ubuntu_clang.sh /work/
-RUN /work/ubuntu_clang.sh
-
-COPY install/ubuntu_tvm.sh /work/
-RUN /work/ubuntu_tvm.sh
-
-COPY install/ubuntu_llvm.sh /work/
-RUN /work/ubuntu_llvm.sh
-
-COPY install/ubuntu_caffe.sh /work/
-RUN /work/ubuntu_caffe.sh
-
-COPY install/ubuntu_onnx.sh /work/
-RUN /work/ubuntu_onnx.sh
-
-COPY install/ubuntu_docs.sh /work/
-COPY install/requirements /work/
-RUN /work/ubuntu_docs.sh
-
-COPY install/ubuntu_tutorials.sh /work/
-RUN /work/ubuntu_tutorials.sh
-
-# Always last
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
diff --git a/ci/docker/Dockerfile.build.ubuntu_gpu_cu101
b/ci/docker/Dockerfile.build.ubuntu_gpu_cu101
deleted file mode 100644
index 5f038b437d..0000000000
--- a/ci/docker/Dockerfile.build.ubuntu_gpu_cu101
+++ /dev/null
@@ -1,75 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to run MXNet on Ubuntu 18.04 for GPU
-
-FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04
-
-WORKDIR /work/deps
-
-COPY install/ubuntu_core.sh /work/
-RUN /work/ubuntu_core.sh
-
-COPY install/deb_ubuntu_ccache.sh /work/
-RUN /work/deb_ubuntu_ccache.sh
-
-COPY install/ubuntu_python.sh /work/
-COPY install/requirements /work/
-RUN /work/ubuntu_python.sh
-
-COPY install/ubuntu_scala.sh /work/
-COPY install/sbt.gpg /work/
-RUN /work/ubuntu_scala.sh
-
-COPY install/ubuntu_r.sh /work/
-RUN /work/ubuntu_r.sh
-
-COPY install/ubuntu_perl.sh /work/
-RUN /work/ubuntu_perl.sh
-
-COPY install/ubuntu_clang.sh /work/
-RUN /work/ubuntu_clang.sh
-
-COPY install/ubuntu_tvm.sh /work/
-RUN /work/ubuntu_tvm.sh
-
-COPY install/ubuntu_llvm.sh /work/
-RUN /work/ubuntu_llvm.sh
-
-COPY install/ubuntu_caffe.sh /work/
-RUN /work/ubuntu_caffe.sh
-
-COPY install/ubuntu_onnx.sh /work/
-RUN /work/ubuntu_onnx.sh
-
-COPY install/ubuntu_docs.sh /work/
-COPY install/requirements /work/
-RUN /work/ubuntu_docs.sh
-
-COPY install/ubuntu_tutorials.sh /work/
-RUN /work/ubuntu_tutorials.sh
-
-# Always last
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
diff --git a/ci/docker/Dockerfile.build.ubuntu_gpu_cu110
b/ci/docker/Dockerfile.build.ubuntu_gpu_cu110
deleted file mode 100644
index 923c06979d..0000000000
--- a/ci/docker/Dockerfile.build.ubuntu_gpu_cu110
+++ /dev/null
@@ -1,43 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to run MXNet on Ubuntu 18.04 for GPU
-
-FROM nvidia/cuda:11.0-cudnn8-devel-ubuntu18.04
-
-WORKDIR /work/deps
-
-COPY install/ubuntu_core.sh /work/
-RUN /work/ubuntu_core.sh
-
-COPY install/deb_ubuntu_ccache.sh /work/
-RUN /work/deb_ubuntu_ccache.sh
-
-COPY install/ubuntu_python.sh /work/
-COPY install/requirements /work/
-RUN /work/ubuntu_python.sh
-
-# Always last
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
diff --git a/ci/docker/Dockerfile.build.ubuntu_gpu_cu112
b/ci/docker/Dockerfile.build.ubuntu_gpu_cu112
deleted file mode 100644
index f45ceed734..0000000000
--- a/ci/docker/Dockerfile.build.ubuntu_gpu_cu112
+++ /dev/null
@@ -1,43 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to run MXNet on Ubuntu 18.04 for GPU
-
-FROM nvidia/cuda:11.2.1-cudnn8-devel-ubuntu18.04
-
-WORKDIR /work/deps
-
-COPY install/ubuntu_core.sh /work/
-RUN /work/ubuntu_core.sh
-
-COPY install/deb_ubuntu_ccache.sh /work/
-RUN /work/deb_ubuntu_ccache.sh
-
-COPY install/ubuntu_python.sh /work/
-COPY install/requirements /work/
-RUN /work/ubuntu_python.sh
-
-# Always last
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
diff --git a/ci/docker/Dockerfile.build.ubuntu_gpu_tensorrt
b/ci/docker/Dockerfile.build.ubuntu_gpu_tensorrt
index 5e9af00266..0b7584c0db 100644
--- a/ci/docker/Dockerfile.build.ubuntu_gpu_tensorrt
+++ b/ci/docker/Dockerfile.build.ubuntu_gpu_tensorrt
@@ -26,11 +26,18 @@ RUN apt-key adv --fetch-keys
https://developer.download.nvidia.com/compute/cuda/
COPY install/ubuntu_core.sh /work/
RUN /work/ubuntu_core.sh
+
+COPY install/ubuntu_cmake.sh /work/
+RUN /work/ubuntu_cmake.sh
+
COPY install/deb_ubuntu_ccache.sh /work/
RUN /work/deb_ubuntu_ccache.sh
+
COPY install/ubuntu_python.sh /work/
-COPY install/requirements /work/
RUN /work/ubuntu_python.sh
+COPY install/requirements /work/
+RUN pip3 install -r /work/requirements
+
COPY install/tensorrt.sh /work
RUN /work/tensorrt.sh
diff --git a/ci/docker/Dockerfile.build.ubuntu_nightly_cpu
b/ci/docker/Dockerfile.build.ubuntu_nightly_cpu
index 65ea937c45..740e0143ff 100644
--- a/ci/docker/Dockerfile.build.ubuntu_nightly_cpu
+++ b/ci/docker/Dockerfile.build.ubuntu_nightly_cpu
@@ -25,13 +25,12 @@ WORKDIR /work/deps
COPY install/ubuntu_core.sh /work/
RUN /work/ubuntu_core.sh
+COPY install/ubuntu_cmake.sh /work/
+RUN /work/ubuntu_cmake.sh
+
COPY install/deb_ubuntu_ccache.sh /work/
RUN /work/deb_ubuntu_ccache.sh
-COPY install/ubuntu_python.sh /work/
-COPY install/requirements /work/
-RUN /work/ubuntu_python.sh
-
COPY install/ubuntu_scala.sh /work/
COPY install/sbt.gpg /work/
RUN /work/ubuntu_scala.sh
@@ -48,6 +47,11 @@ RUN /work/ubuntu_clang.sh
COPY install/ubuntu_caffe.sh /work/
RUN /work/ubuntu_caffe.sh
+COPY install/ubuntu_python.sh /work/
+RUN /work/ubuntu_python.sh
+COPY install/requirements /work/
+RUN pip3 install -r /work/requirements
+
COPY install/ubuntu_onnx.sh /work/
RUN /work/ubuntu_onnx.sh
diff --git a/ci/docker/Dockerfile.build.ubuntu_nightly_gpu
b/ci/docker/Dockerfile.build.ubuntu_nightly_gpu
index c2d255bfe2..4ea27b3212 100644
--- a/ci/docker/Dockerfile.build.ubuntu_nightly_gpu
+++ b/ci/docker/Dockerfile.build.ubuntu_nightly_gpu
@@ -18,20 +18,19 @@
#
# Dockerfile to run MXNet on Ubuntu 18.04 for GPU
-FROM nvidia/cuda:10.1-devel-ubuntu18.04
+FROM nvidia/cuda:11.7.1-cudnn8-devel-ubuntu18.04
WORKDIR /work/deps
COPY install/ubuntu_core.sh /work/
RUN /work/ubuntu_core.sh
+COPY install/ubuntu_cmake.sh /work/
+RUN /work/ubuntu_cmake.sh
+
COPY install/deb_ubuntu_ccache.sh /work/
RUN /work/deb_ubuntu_ccache.sh
-COPY install/ubuntu_python.sh /work/
-COPY install/requirements /work/
-RUN /work/ubuntu_python.sh
-
COPY install/ubuntu_scala.sh /work/
COPY install/sbt.gpg /work/
RUN /work/ubuntu_scala.sh
@@ -45,15 +44,20 @@ RUN /work/ubuntu_perl.sh
COPY install/ubuntu_clang.sh /work/
RUN /work/ubuntu_clang.sh
-COPY install/ubuntu_tvm.sh /work/
-RUN /work/ubuntu_tvm.sh
-
COPY install/ubuntu_llvm.sh /work/
RUN /work/ubuntu_llvm.sh
COPY install/ubuntu_caffe.sh /work/
RUN /work/ubuntu_caffe.sh
+COPY install/ubuntu_python.sh /work/
+RUN /work/ubuntu_python.sh
+COPY install/requirements /work/
+RUN pip3 install -r /work/requirements
+
+COPY install/ubuntu_tvm.sh /work/
+RUN /work/ubuntu_tvm.sh
+
COPY install/ubuntu_onnx.sh /work/
RUN /work/ubuntu_onnx.sh
@@ -67,11 +71,6 @@ RUN /work/ubuntu_tutorials.sh
COPY install/ubuntu_nightly_tests.sh /work/
RUN /work/ubuntu_nightly_tests.sh
-ENV CUDA_VERSION=10.1.105
-ENV CUDNN_VERSION=7.6.5.32
-COPY install/ubuntu_cudnn.sh /work/
-RUN /work/ubuntu_cudnn.sh
-
# Always last
ARG USER_ID=0
ARG GROUP_ID=0
diff --git a/ci/docker/Dockerfile.publish.test.centos7_cpu
b/ci/docker/Dockerfile.publish.test.centos7_cpu
deleted file mode 100644
index 7d28445297..0000000000
--- a/ci/docker/Dockerfile.publish.test.centos7_cpu
+++ /dev/null
@@ -1,38 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to build and run MXNet on CentOS 7 for CPU
-
-FROM centos:7
-
-WORKDIR /work/deps
-
-COPY install/centos7_base.sh /work/
-RUN /work/centos7_base.sh
-
-COPY install/centos7_scala.sh /work/
-RUN /work/centos7_scala.sh
-
-ARG USER_ID=0
-COPY install/centos7_adduser.sh /work/
-RUN /work/centos7_adduser.sh
-
-ENV PYTHONPATH=./python/
-WORKDIR /work/mxnet
-
-COPY runtime_functions.sh /work/
diff --git a/ci/docker/Dockerfile.publish.test.centos7_gpu
b/ci/docker/Dockerfile.publish.test.centos7_gpu
deleted file mode 100644
index e7f5846831..0000000000
--- a/ci/docker/Dockerfile.publish.test.centos7_gpu
+++ /dev/null
@@ -1,38 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to build and run MXNet on CentOS 7 for CPU
-
-FROM nvidia/cuda:9.2-cudnn7-devel-centos7
-
-WORKDIR /work/deps
-
-COPY install/centos7_base.sh /work/
-RUN /work/centos7_base.sh
-
-COPY install/centos7_scala.sh /work/
-RUN /work/centos7_scala.sh
-
-ARG USER_ID=0
-COPY install/centos7_adduser.sh /work/
-RUN /work/centos7_adduser.sh
-
-ENV PYTHONPATH=./python/
-WORKDIR /work/mxnet
-
-COPY runtime_functions.sh /work/
diff --git a/ci/docker/Dockerfile.publish.test.ubuntu1404_cpu
b/ci/docker/Dockerfile.publish.test.ubuntu1404_cpu
deleted file mode 100644
index 0358376865..0000000000
--- a/ci/docker/Dockerfile.publish.test.ubuntu1404_cpu
+++ /dev/null
@@ -1,39 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to build and run MXNet on Ubuntu 14.04 for CPU
-
-FROM ubuntu:14.04
-
-WORKDIR /work/deps
-
-COPY install/ubuntu_base.sh /work/
-RUN /work/ubuntu_base.sh
-
-COPY install/ubuntu_scala.sh /work/
-RUN /work/ubuntu_scala.sh
-
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
-ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
diff --git a/ci/docker/Dockerfile.publish.test.ubuntu1404_gpu
b/ci/docker/Dockerfile.publish.test.ubuntu1404_gpu
deleted file mode 100644
index 854dd68a63..0000000000
--- a/ci/docker/Dockerfile.publish.test.ubuntu1404_gpu
+++ /dev/null
@@ -1,40 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to run MXNet on Ubuntu 14.04 for GPU
-
-# Use CPU with setup_gpu script
-FROM ubuntu:14.04
-
-WORKDIR /work/deps
-
-COPY install/ubuntu_base.sh /work/
-RUN /work/ubuntu_base.sh
-
-COPY install/ubuntu_scala.sh /work/
-RUN /work/ubuntu_scala.sh
-
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
-ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
diff --git a/ci/docker/Dockerfile.publish.test.ubuntu1604_cpu
b/ci/docker/Dockerfile.publish.test.ubuntu1604_cpu
deleted file mode 100644
index bbb7b6a0d7..0000000000
--- a/ci/docker/Dockerfile.publish.test.ubuntu1604_cpu
+++ /dev/null
@@ -1,39 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to build and run MXNet on Ubuntu 16.04 for CPU
-
-FROM ubuntu:16.04
-
-WORKDIR /work/deps
-
-COPY install/ubuntu_base.sh /work/
-RUN /work/ubuntu_base.sh
-
-COPY install/ubuntu_scala.sh /work/
-RUN /work/ubuntu_scala.sh
-
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
-ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
diff --git a/ci/docker/Dockerfile.publish.test.ubuntu1604_gpu
b/ci/docker/Dockerfile.publish.test.ubuntu1604_gpu
deleted file mode 100644
index 660461dc0c..0000000000
--- a/ci/docker/Dockerfile.publish.test.ubuntu1604_gpu
+++ /dev/null
@@ -1,39 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to run MXNet on Ubuntu 16.04 for GPU
-
-FROM nvidia/cuda:9.2-cudnn7-devel-ubuntu16.04
-
-WORKDIR /work/deps
-
-COPY install/ubuntu_base.sh /work/
-RUN /work/ubuntu_base.sh
-
-COPY install/ubuntu_scala.sh /work/
-RUN /work/ubuntu_scala.sh
-
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
-ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
diff --git a/ci/docker/Dockerfile.publish.test.ubuntu1804_cpu
b/ci/docker/Dockerfile.publish.test.ubuntu1804_cpu
deleted file mode 100644
index e3a8c193f2..0000000000
--- a/ci/docker/Dockerfile.publish.test.ubuntu1804_cpu
+++ /dev/null
@@ -1,41 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to build and run MXNet on Ubuntu 18.04 for CPU
-
-FROM ubuntu:18.04
-
-WORKDIR /work/deps
-
-ENV DEBIAN_FRONTEND noninteractive
-
-COPY install/ubuntu_base.sh /work/
-RUN /work/ubuntu_base.sh
-
-COPY install/ubuntu_scala.sh /work/
-RUN /work/ubuntu_scala.sh
-
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
-ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
diff --git a/ci/docker/Dockerfile.publish.test.ubuntu1804_gpu
b/ci/docker/Dockerfile.publish.test.ubuntu1804_gpu
deleted file mode 100644
index 99f7e0d3ef..0000000000
--- a/ci/docker/Dockerfile.publish.test.ubuntu1804_gpu
+++ /dev/null
@@ -1,41 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to run MXNet on Ubuntu 18.04 for GPU
-
-FROM nvidia/cuda:9.2-cudnn7-devel-ubuntu18.04
-
-WORKDIR /work/deps
-
-ENV DEBIAN_FRONTEND noninteractive
-
-COPY install/ubuntu_base.sh /work/
-RUN /work/ubuntu_base.sh
-
-COPY install/ubuntu_scala.sh /work/
-RUN /work/ubuntu_scala.sh
-
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
-ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
diff --git a/ci/docker/Dockerfile.publish.ubuntu1404_cpu
b/ci/docker/Dockerfile.publish.ubuntu1404_cpu
deleted file mode 100644
index 8ccc41b214..0000000000
--- a/ci/docker/Dockerfile.publish.ubuntu1404_cpu
+++ /dev/null
@@ -1,39 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to build and run MXNet on Ubuntu 14.04 for CPU
-
-FROM ubuntu:14.04
-
-WORKDIR /work/deps
-
-COPY install/ubuntu_publish.sh /work/
-RUN /work/ubuntu_publish.sh
-
-COPY install/ubuntu_binutils.sh /work/
-RUN /work/ubuntu_binutils.sh
-
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
-ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
diff --git a/ci/docker/Dockerfile.publish.ubuntu1404_gpu
b/ci/docker/Dockerfile.publish.ubuntu1404_gpu
deleted file mode 100644
index 3a005cadec..0000000000
--- a/ci/docker/Dockerfile.publish.ubuntu1404_gpu
+++ /dev/null
@@ -1,39 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to run MXNet on Ubuntu 14.04 for GPU
-
-FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu14.04
-
-WORKDIR /work/deps
-
-COPY install/ubuntu_publish.sh /work/
-RUN /work/ubuntu_publish.sh
-
-COPY install/ubuntu_binutils.sh /work/
-RUN /work/ubuntu_binutils.sh
-
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
-ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
diff --git a/ci/docker/Dockerfile.publish.ubuntu1604_cpu
b/ci/docker/Dockerfile.publish.ubuntu1604_cpu
deleted file mode 100644
index e5898b66c1..0000000000
--- a/ci/docker/Dockerfile.publish.ubuntu1604_cpu
+++ /dev/null
@@ -1,44 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to build and run MXNet on Ubuntu 16.04 for CPU
-
-FROM ubuntu:16.04
-
-WORKDIR /work/deps
-
-COPY install/ubuntu_base.sh /work/
-RUN /work/ubuntu_base.sh
-
-COPY install/ubuntu_python.sh /work/
-COPY install/requirements /work/
-RUN /work/ubuntu_python.sh
-
-COPY install/ubuntu_scala.sh /work/
-COPY install/sbt.gpg /work/
-RUN /work/ubuntu_scala.sh
-
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
-ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
diff --git a/ci/docker/Dockerfile.publish.ubuntu1604_gpu
b/ci/docker/Dockerfile.publish.ubuntu1604_gpu
deleted file mode 100644
index 0bd8b8259b..0000000000
--- a/ci/docker/Dockerfile.publish.ubuntu1604_gpu
+++ /dev/null
@@ -1,44 +0,0 @@
-# -*- mode: dockerfile -*-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# Dockerfile to run MXNet on Ubuntu 16.04 for GPU
-
-FROM nvidia/cuda:9.2-cudnn7-devel-ubuntu16.04
-
-WORKDIR /work/deps
-
-COPY install/ubuntu_base.sh /work/
-RUN /work/ubuntu_base.sh
-
-COPY install/ubuntu_python.sh /work/
-COPY install/requirements /work/
-RUN /work/ubuntu_python.sh
-
-COPY install/ubuntu_scala.sh /work/
-COPY install/sbt.gpg /work/
-RUN /work/ubuntu_scala.sh
-
-ARG USER_ID=0
-ARG GROUP_ID=0
-COPY install/ubuntu_adduser.sh /work/
-RUN /work/ubuntu_adduser.sh
-
-COPY runtime_functions.sh /work/
-
-WORKDIR /work/mxnet
-ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
diff --git a/ci/docker/docker-compose.yml b/ci/docker/docker-compose.yml
index f5f49a2460..02d988b595 100644
--- a/ci/docker/docker-compose.yml
+++ b/ci/docker/docker-compose.yml
@@ -35,8 +35,6 @@ services:
build:
context: .
dockerfile: Dockerfile.build.centos7
- # Use "base" target declared in Dockerfile.build.centos7 as
"build.centos7_cpu:latest"
- target: base
args:
# BASE_IMAGE is used to dynamically specify the FROM image in
Dockerfile.build.centos7
BASE_IMAGE: centos:7
@@ -47,7 +45,6 @@ services:
build:
context: .
dockerfile: Dockerfile.build.centos7
- target: base
args:
BASE_IMAGE: nvidia/cuda:10.0-cudnn7-devel-centos7
cache_from:
@@ -57,7 +54,6 @@ services:
build:
context: .
dockerfile: Dockerfile.build.centos7
- target: base
args:
BASE_IMAGE: nvidia/cuda:10.1-cudnn7-devel-centos7
cache_from:
@@ -67,7 +63,6 @@ services:
build:
context: .
dockerfile: Dockerfile.build.centos7
- target: base
args:
BASE_IMAGE: nvidia/cuda:10.2-cudnn8-devel-centos7
cache_from:
@@ -77,27 +72,42 @@ services:
build:
context: .
dockerfile: Dockerfile.build.centos7
- target: base
args:
BASE_IMAGE: nvidia/cuda:11.0.3-cudnn8-devel-centos7
cache_from:
- ${DOCKER_CACHE_REGISTRY}:build.centos7_gpu_cu110
+ centos7_gpu_cu111:
+ image: ${DOCKER_CACHE_REGISTRY}:build.centos7_gpu_cu111
+ build:
+ context: .
+ dockerfile: Dockerfile.build.centos7
+ args:
+ BASE_IMAGE: nvidia/cuda:11.1.1-cudnn8-devel-centos7
+ cache_from:
+ - ${DOCKER_CACHE_REGISTRY}:build.centos7_gpu_cu111
centos7_gpu_cu112:
image: ${DOCKER_CACHE_REGISTRY}:build.centos7_gpu_cu112
build:
context: .
dockerfile: Dockerfile.build.centos7
- target: base
args:
BASE_IMAGE: nvidia/cuda:11.2.2-cudnn8-devel-centos7
cache_from:
- ${DOCKER_CACHE_REGISTRY}:build.centos7_gpu_cu112
+ centos7_gpu_cu113:
+ image: ${DOCKER_CACHE_REGISTRY}:build.centos7_gpu_cu113
+ build:
+ context: .
+ dockerfile: Dockerfile.build.centos7
+ args:
+ BASE_IMAGE: nvidia/cuda:11.3.1-cudnn8-devel-centos7
+ cache_from:
+ - ${DOCKER_CACHE_REGISTRY}:build.centos7_gpu_cu113
centos7_gpu_cu114:
image: ${DOCKER_CACHE_REGISTRY}:build.centos7_gpu_cu114
build:
context: .
dockerfile: Dockerfile.build.centos7
- target: base
args:
BASE_IMAGE: nvidia/cuda:11.4.3-cudnn8-devel-centos7
cache_from:
@@ -107,7 +117,6 @@ services:
build:
context: .
dockerfile: Dockerfile.build.centos7
- target: base
args:
BASE_IMAGE: nvidia/cuda:11.5.2-cudnn8-devel-centos7
cache_from:
@@ -117,7 +126,6 @@ services:
build:
context: .
dockerfile: Dockerfile.build.centos7
- target: base
args:
BASE_IMAGE: nvidia/cuda:11.6.2-cudnn8-devel-centos7
cache_from:
@@ -127,8 +135,95 @@ services:
build:
context: .
dockerfile: Dockerfile.build.centos7
- target: base
args:
BASE_IMAGE: nvidia/cuda:11.7.1-cudnn8-devel-centos7
cache_from:
- ${DOCKER_CACHE_REGISTRY}:build.centos7_gpu_cu117
+ ubuntu_cpu:
+ image: ${DOCKER_CACHE_REGISTRY}:build.ubuntu_cpu
+ build:
+ context: .
+ dockerfile: Dockerfile.build.ubuntu_cpu
+ cache_from:
+ - ${DOCKER_CACHE_REGISTRY}:build.ubuntu_cpu
+ ubuntu_gpu_cu100:
+ image: ${DOCKER_CACHE_REGISTRY}:build.ubuntu_gpu_cu100
+ build:
+ context: .
+ dockerfile: Dockerfile.build.ubuntu_gpu
+ args:
+ BASE_IMAGE: nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04
+ cache_from:
+ - ${DOCKER_CACHE_REGISTRY}:build.ubuntu_gpu_cu100
+ ubuntu_gpu_cu101:
+ image: ${DOCKER_CACHE_REGISTRY}:build.ubuntu_gpu_cu101
+ build:
+ context: .
+ dockerfile: Dockerfile.build.ubuntu_gpu
+ args:
+ BASE_IMAGE: nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04
+ cache_from:
+ - ${DOCKER_CACHE_REGISTRY}:build.ubuntu_gpu_cu101
+ ubuntu_gpu_cu102:
+ image: ${DOCKER_CACHE_REGISTRY}:build.ubuntu_gpu_cu102
+ build:
+ context: .
+ dockerfile: Dockerfile.build.ubuntu_gpu
+ args:
+ BASE_IMAGE: nvidia/cuda:10.2-cudnn8-devel-ubuntu18.04
+ cache_from:
+ - ${DOCKER_CACHE_REGISTRY}:build.ubuntu_gpu_cu102
+ ubuntu_gpu_cu110:
+ image: ${DOCKER_CACHE_REGISTRY}:build.ubuntu_gpu_cu110
+ build:
+ context: .
+ dockerfile: Dockerfile.build.ubuntu_gpu
+ args:
+ BASE_IMAGE: nvidia/cuda:11.0.3-cudnn8-devel-ubuntu18.04
+ cache_from:
+ - ${DOCKER_CACHE_REGISTRY}:build.ubuntu_gpu_cu110
+ ubuntu_gpu_cu112:
+ image: ${DOCKER_CACHE_REGISTRY}:build.ubuntu_gpu_cu112
+ build:
+ context: .
+ dockerfile: Dockerfile.build.ubuntu_gpu
+ args:
+ BASE_IMAGE: nvidia/cuda:11.2.2-cudnn8-devel-ubuntu18.04
+ cache_from:
+ - ${DOCKER_CACHE_REGISTRY}:build.ubuntu_gpu_cu112
+ ubuntu_gpu_cu114:
+ image: ${DOCKER_CACHE_REGISTRY}:build.ubuntu_gpu_cu114
+ build:
+ context: .
+ dockerfile: Dockerfile.build.ubuntu_gpu
+ args:
+ BASE_IMAGE: nvidia/cuda:11.4.3-cudnn8-devel-ubuntu18.04
+ cache_from:
+ - ${DOCKER_CACHE_REGISTRY}:build.ubuntu_gpu_cu114
+ ubuntu_gpu_cu115:
+ image: ${DOCKER_CACHE_REGISTRY}:build.ubuntu_gpu_cu115
+ build:
+ context: .
+ dockerfile: Dockerfile.build.ubuntu_gpu
+ args:
+ BASE_IMAGE: nvidia/cuda:11.5.2-cudnn8-devel-ubuntu18.04
+ cache_from:
+ - ${DOCKER_CACHE_REGISTRY}:build.ubuntu_gpu_cu115
+ ubuntu_gpu_cu116:
+ image: ${DOCKER_CACHE_REGISTRY}:build.ubuntu_gpu_cu116
+ build:
+ context: .
+ dockerfile: Dockerfile.build.ubuntu_gpu
+ args:
+ BASE_IMAGE: nvidia/cuda:11.6.2-cudnn8-devel-ubuntu18.04
+ cache_from:
+ - ${DOCKER_CACHE_REGISTRY}:build.ubuntu_gpu_cu116
+ ubuntu_gpu_cu117:
+ image: ${DOCKER_CACHE_REGISTRY}:build.ubuntu_gpu_cu117
+ build:
+ context: .
+ dockerfile: Dockerfile.build.ubuntu_gpu
+ args:
+ BASE_IMAGE: nvidia/cuda:11.7.1-cudnn8-devel-ubuntu18.04
+ cache_from:
+ - ${DOCKER_CACHE_REGISTRY}:build.ubuntu_gpu_cu117
diff --git a/ci/docker/install/centos7_base.sh
b/ci/docker/install/centos7_base.sh
index 3b84aeb57b..9d66e427a0 100755
--- a/ci/docker/install/centos7_base.sh
+++ b/ci/docker/install/centos7_base.sh
@@ -24,10 +24,4 @@ set -ex
# Multipackage installation does not fail in yum
yum -y install epel-release
-yum -y install git
-yum -y install wget
-yum -y install make
-yum -y install cmake
-yum -y install unzip
-yum -y install ninja-build
-yum -y install gcc-gfortran
+yum -y install git wget make unzip ninja-build gcc-gfortran
diff --git a/ci/docker/install/centos7_core.sh
b/ci/docker/install/centos7_core.sh
index f9aa7a718e..4ff2eabd7c 100755
--- a/ci/docker/install/centos7_core.sh
+++ b/ci/docker/install/centos7_core.sh
@@ -25,31 +25,23 @@ set -ex
# Multipackage installation does not fail in yum
yum -y install ca-certificates
yum -y install epel-release
-yum -y install git
-yum -y install wget
-yum -y install atlas-devel # Provide clbas headerfiles
-yum -y install openblas-devel
-yum -y install lapack-devel
-yum -y install opencv-devel
-yum -y install protobuf-compiler
-yum -y install protobuf-devel
-yum -y install zeromq-devel
-yum -y install openssl-devel
-yum -y install gcc-c++-4.8.*
-yum -y install make
-yum -y install wget
-yum -y install unzip
-yum -y install ninja-build
-yum -y install automake
-yum -y install patchelf
-yum -y install nasm
-yum -y install libtool
-yum -y install dpkg-dev
+yum -y install git \
+ wget \
+ atlas-devel \
+ openblas-devel \
+ lapack-devel \
+ opencv-devel \
+ protobuf-compiler \
+ protobuf-devel \
+ zeromq-devel \
+ openssl-devel \
+ gcc-c++-4.8.* \
+ make \
+ unzip \
+ ninja-build \
+ automake \
+ patchelf \
+ nasm \
+ libtool \
+ dpkg-dev
-# CMake 3.13.2+ is required
-mkdir /opt/cmake && cd /opt/cmake
-wget -nv https://cmake.org/files/v3.13/cmake-3.13.5-Linux-x86_64.sh
-sh cmake-3.13.5-Linux-x86_64.sh --prefix=/opt/cmake --skip-license
-ln -s /opt/cmake/bin/cmake /usr/local/bin/cmake
-rm cmake-3.13.5-Linux-x86_64.sh
-cmake --version
diff --git a/ci/docker/install/centos7_python.sh
b/ci/docker/install/centos7_python.sh
index edaff09537..1856f68061 100755
--- a/ci/docker/install/centos7_python.sh
+++ b/ci/docker/install/centos7_python.sh
@@ -35,6 +35,3 @@ make install
cd ..
rm -rf Python-$PYTHON_VERSION*
pip3 install --upgrade pip setuptools wheel
-if [[ -f /work/requirements ]]; then
- pip3 install -r /work/requirements
-fi
diff --git a/ci/docker/install/requirements b/ci/docker/install/requirements
index 468a7bb456..8e27a1f31c 100644
--- a/ci/docker/install/requirements
+++ b/ci/docker/install/requirements
@@ -22,7 +22,7 @@ boto3==1.9.229
cpplint==1.3.0
Cython
decorator==4.4.0
-h5py<3
+h5py<3; platform_machine != 'aarch64'
mock==2.0.0
nose==1.3.7
nose-timer==0.7.3
diff --git a/ci/docker/install/requirements_aarch64
b/ci/docker/install/requirements_aarch64
deleted file mode 100644
index 6f45950ded..0000000000
--- a/ci/docker/install/requirements_aarch64
+++ /dev/null
@@ -1,34 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-# build and install are separated so changes to build don't invalidate
-# the whole docker cache for the image
-
-boto3==1.9.229
-cpplint==1.3.0
-Cython
-decorator==4.4.0
-mock==2.0.0
-nose==1.3.7
-nose-timer==0.7.3
-pylint==2.3.1 # pylint and astroid need to be aligned
-astroid==2.3.3 # pylint and astroid need to be aligned
-requests<2.19.0,>=2.18.4
-setuptools
-coverage
-wheel
-packaging
diff --git a/ci/docker/install/ubuntu_core.sh b/ci/docker/install/ubuntu_core.sh
index 3cf48b55bf..f49ac218a2 100755
--- a/ci/docker/install/ubuntu_core.sh
+++ b/ci/docker/install/ubuntu_core.sh
@@ -68,11 +68,3 @@ if [[ "$VERSION_ID" != "16.04" ]]; then
apt-get install -y libturbojpeg0-dev
fi
-
-# CMake 3.13.2+ is required
-mkdir /opt/cmake && cd /opt/cmake
-wget -nv https://cmake.org/files/v3.13/cmake-3.13.5-Linux-x86_64.sh
-sh cmake-3.13.5-Linux-x86_64.sh --prefix=/opt/cmake --skip-license
-ln -s /opt/cmake/bin/cmake /usr/local/bin/cmake
-rm cmake-3.13.5-Linux-x86_64.sh
-cmake --version
diff --git a/ci/docker/install/ubuntu_publish.sh
b/ci/docker/install/ubuntu_publish.sh
deleted file mode 100755
index b5a8906cc2..0000000000
--- a/ci/docker/install/ubuntu_publish.sh
+++ /dev/null
@@ -1,96 +0,0 @@
-#!/usr/bin/env bash
-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# Build on Ubuntu 14.04 LTS for LINUX CPU/GPU
-set -ex
-
-# replace https with http to force apt-get update to use http
-# nvidia-docker no longer supports ubuntu 14.04
-# refer https://github.com/apache/incubator-mxnet/issues/18005
-sudo sed -i 's/https/http/g' /etc/apt/sources.list.d/*.list
-apt-get update
-apt-get install -y software-properties-common
-add-apt-repository ppa:ubuntu-toolchain-r/test -y
-add-apt-repository ppa:openjdk-r/ppa -y # Java lib
-apt-get update
-apt-get install -y git \
- cmake3 \
- ninja-build \
- libcurl4-openssl-dev \
- unzip \
- gcc-4.8 \
- g++-4.8 \
- gfortran \
- gfortran-4.8 \
- binutils \
- nasm \
- libtool \
- curl \
- wget \
- sudo \
- gnupg \
- gnupg2 \
- gnupg-agent \
- pandoc \
- python3-pip \
- automake \
- pkg-config \
- openjdk-8-jdk
-
-# disable expired cert that causes download issues with let's encrypt certs
-sed -i 's/mozilla\/DST_Root_CA_X3.crt/!mozilla\/DST_Root_CA_X3.crt/g'
/etc/ca-certificates.conf
-update-ca-certificates
-
-curl -o apache-maven-3.3.9-bin.tar.gz -L
http://www.eu.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
\
- || curl -o apache-maven-3.3.9-bin.tar.gz -L
https://search.maven.org/remotecontent?filepath=org/apache/maven/apache-maven/3.3.9/apache-maven-3.3.9-bin.tar.gz
-
-tar xzf apache-maven-3.3.9-bin.tar.gz
-mkdir /usr/local/maven
-mv apache-maven-3.3.9/ /usr/local/maven/
-update-alternatives --install /usr/bin/mvn mvn
/usr/local/maven/apache-maven-3.3.9/bin/mvn 1
-update-ca-certificates -f
-
-# patchelf available starting Ubuntu 16.04; compile from source for Ubuntu
14.04
-mkdir /usr/local/patchelf
-cd /usr/local/patchelf
-curl -L -o patchelf-0.10.tar.gz
https://github.com/NixOS/patchelf/archive/0.10.tar.gz
-tar xzf patchelf-0.10.tar.gz
-cd /usr/local/patchelf/patchelf-0.10
-./bootstrap.sh
-./configure
-make
-sudo make install
-cd /
-
-apt-get install -y python python-pip python3 python3-pip
-
-# the version of the pip shipped with ubuntu may be too lower, install a
recent version here
-# Restrict pip version to <19 due to use of Python 3.4 on Ubuntu 14.04
-python3 -m pip install --upgrade 'pip<19'
-
-# Allow numpy version as advanced as 1.19.5 to avoid CVE-2021-41495 and
CVE-2021-41496 affecting <1.19.1.
-python3 -m pip install --upgrade --ignore-installed nose cpplint==1.3.0
pylint==2.3.1 'numpy>=1.16.0,<1.20.0' nose-timer 'requests<2.19.0,>=2.18.4'
'h5py<3' scipy==1.0.1 boto3 packaging
-
-# CMake 3.13.2+ is required
-mkdir /opt/cmake && cd /opt/cmake
-wget -nv https://cmake.org/files/v3.13/cmake-3.13.5-Linux-x86_64.sh
-sh cmake-3.13.5-Linux-x86_64.sh --prefix=/opt/cmake --skip-license
-ln -s /opt/cmake/bin/cmake /usr/local/bin/cmake
-rm cmake-3.13.5-Linux-x86_64.sh
-cmake --version
diff --git a/ci/docker/install/ubuntu_python.sh
b/ci/docker/install/ubuntu_python.sh
index bdce2ab862..a9e3df0ce5 100755
--- a/ci/docker/install/ubuntu_python.sh
+++ b/ci/docker/install/ubuntu_python.sh
@@ -21,16 +21,17 @@
# the whole docker cache for the image
set -ex
-# install libraries for mxnet's python package on ubuntu
-apt-get update || true
-apt-get install -y software-properties-common
-add-apt-repository -y ppa:deadsnakes/ppa
-apt-get update || true
-apt-get install -y python3.7-dev python3.7-distutils virtualenv wget
-# setup symlink in /usr/local/bin to override python3 version
-ln -sf /usr/bin/python3.7 /usr/local/bin/python3
-# the version of the pip shipped with ubuntu may be too lower, install a
recent version here
-wget -nv https://bootstrap.pypa.io/get-pip.py
-/usr/local/bin/python3 get-pip.py
-/usr/local/bin/python3 -m pip install -r /work/requirements
+apt-get update
+apt-get install -y make wget libssl-dev libbz2-dev libffi-dev libhdf5-dev
libsqlite3-dev
+
+PYTHON_VERSION=3.8.12
+wget -q
https://www.python.org/ftp/python/$PYTHON_VERSION/Python-$PYTHON_VERSION.tgz
+tar -xzf Python-$PYTHON_VERSION.tgz
+cd Python-$PYTHON_VERSION
+./configure --prefix=/usr/local
+make -j $(nproc)
+make install
+cd ..
+rm -rf Python-$PYTHON_VERSION*
+pip3 install --upgrade pip setuptools wheel
diff --git a/ci/docker/install/ubuntu_tutorials.sh
b/ci/docker/install/ubuntu_tutorials.sh
index 469df6190e..4757c69534 100755
--- a/ci/docker/install/ubuntu_tutorials.sh
+++ b/ci/docker/install/ubuntu_tutorials.sh
@@ -22,7 +22,7 @@
set -ex
apt-get update || true
-apt-get install graphviz python-opencv
+apt-get install graphviz
# sckit-learn past version 0.20 does not support python version 2 and 3.4
pip3 install jupyter matplotlib Pillow opencv-python scikit-learn
graphviz==0.8.4 tqdm mxboard scipy gluoncv
diff --git a/ci/docker/install/ubuntu_tvm.sh b/ci/docker/install/ubuntu_tvm.sh
index 2ee4e534ed..ba4d2353e0 100755
--- a/ci/docker/install/ubuntu_tvm.sh
+++ b/ci/docker/install/ubuntu_tvm.sh
@@ -17,15 +17,18 @@
# specific language governing permissions and limitations
# under the License.
+set -e
+
# Build and install TVM
cd /tmp
-git clone https://github.com/dmlc/tvm/ --recursive
+git clone https://github.com/dmlc/tvm.git tvm
cd tvm
# This is a stable tag that support MXNet TVM bridge.
# We use this since support for mxnet bridge just checked
# into master and there is yet a version tag
git checkout v0.4
+git submodule update --init --recursive
cp cmake/config.cmake .
echo set\(USE_CUDA /usr/local/cuda\) >> config.cmake
@@ -36,9 +39,9 @@ echo set\(USE_GRAPH_RUNTIME ON\) >> config.cmake
make -j$(nproc)
cd python
-python setup.py install
+python3 setup.py install
cd -
cd topi/python
-python setup.py install
+python3 setup.py install
cd -
diff --git a/ci/docker/runtime_functions.sh b/ci/docker/runtime_functions.sh
index 006c0a31a7..81377c5ffe 100755
--- a/ci/docker/runtime_functions.sh
+++ b/ci/docker/runtime_functions.sh
@@ -705,7 +705,7 @@ build_ubuntu_cpu_mkldnn_mkl() {
}
build_ubuntu_gpu() {
- build_ubuntu_gpu_cuda101_cudnn7
+ build_ubuntu_gpu_cudnn8
}
build_ubuntu_gpu_tensorrt() {
@@ -771,7 +771,7 @@ build_ubuntu_gpu_mkldnn_nocudnn() {
-j$(nproc)
}
-build_ubuntu_gpu_cuda101_cudnn7() {
+build_ubuntu_gpu_cudnn7() {
set -ex
build_ccache_wrappers
make \
@@ -790,7 +790,7 @@ build_ubuntu_gpu_cuda101_cudnn7() {
make cython PYTHON=python3
}
-build_ubuntu_gpu_cuda110_cudnn8() {
+build_ubuntu_gpu_cudnn8() {
set -ex
build_ccache_wrappers
local CUDA_ARCH="-gencode=arch=compute_52,code=sm_52 \
@@ -807,11 +807,12 @@ build_ubuntu_gpu_cuda110_cudnn8() {
USE_DIST_KVSTORE=1 \
CUDA_ARCH="$CUDA_ARCH" \
USE_SIGNAL_HANDLER=1 \
+ USE_LAPACK_PATH=/usr/lib/x86_64-linux-gnu \
-j$(nproc)
make cython PYTHON=python3
}
-build_ubuntu_gpu_cuda101_cudnn7_mkldnn_cpp_test() {
+build_ubuntu_gpu_cudnn8_mkldnn_cpp_test() {
set -ex
build_ccache_wrappers
make \
@@ -984,10 +985,6 @@ cd_unittest_ubuntu() {
local nose_cmd="nosetests-3.4"
- if [[ ${mxnet_variant} = aarch64_cpu ]]; then
- source /opt/rh/rh-python38/enable
- fi
-
$nose_cmd $NOSE_TIMER_ARGUMENTS --verbose tests/python/unittest
$nose_cmd $NOSE_TIMER_ARGUMENTS --verbose tests/python/quantization
@@ -1043,7 +1040,7 @@ unittest_ubuntu_python3_gpu() {
export MXNET_MKLDNN_DEBUG=0 # Ignored if not present
export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
export MXNET_SUBGRAPH_VERBOSE=0
- export CUDNN_VERSION=${CUDNN_VERSION:-7.0.3}
+ export CUDNN_VERSION=${CUDNN_VERSION:-8.5.0.96}
export MXNET_ENABLE_CYTHON=0
export DMLC_LOG_STACK_TRACE_DEPTH=10
nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit
--xunit-file nosetests_gpu.xml --verbose tests/python/gpu
@@ -1055,7 +1052,7 @@ unittest_ubuntu_python3_gpu_cython() {
export MXNET_MKLDNN_DEBUG=1 # Ignored if not present
export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
export MXNET_SUBGRAPH_VERBOSE=0
- export CUDNN_VERSION=${CUDNN_VERSION:-7.0.3}
+ export CUDNN_VERSION=${CUDNN_VERSION:-8.5.0.96}
export MXNET_ENABLE_CYTHON=1
export MXNET_ENFORCE_CYTHON=1
export DMLC_LOG_STACK_TRACE_DEPTH=10
@@ -1098,19 +1095,7 @@ unittest_ubuntu_python3_quantization_gpu() {
export MXNET_MKLDNN_DEBUG=0 # Ignored if not present
export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
export MXNET_SUBGRAPH_VERBOSE=0
- export CUDNN_VERSION=${CUDNN_VERSION:-7.0.3}
- export MXNET_ENABLE_CYTHON=0
- export DMLC_LOG_STACK_TRACE_DEPTH=10
- nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit
--xunit-file nosetests_quantization_gpu.xml --verbose
tests/python/quantization_gpu
-}
-
-unittest_ubuntu_python3_quantization_gpu_cu110() {
- set -ex
- export PYTHONPATH=./python/
- export MXNET_MKLDNN_DEBUG=0 # Ignored if not present
- export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
- export MXNET_SUBGRAPH_VERBOSE=0
- export CUDNN_VERSION=${CUDNN_VERSION:-8.0.33}
+ export CUDNN_VERSION=${CUDNN_VERSION:-8.5.0.96}
export MXNET_ENABLE_CYTHON=0
export DMLC_LOG_STACK_TRACE_DEPTH=10
nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit
--xunit-file nosetests_quantization_gpu.xml --verbose
tests/python/quantization_gpu
@@ -1251,7 +1236,7 @@ unittest_centos7_cpu() {
unittest_centos7_gpu() {
set -ex
cd /work/mxnet
- export CUDNN_VERSION=${CUDNN_VERSION:-7.0.3}
+ export CUDNN_VERSION=${CUDNN_VERSION:-8.5.0.96}
export DMLC_LOG_STACK_TRACE_DEPTH=10
python3 -m "nose" $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS
--with-xunit --xunit-file nosetests_gpu.xml --verbose tests/python/gpu
}
@@ -2023,9 +2008,6 @@ cd_package_pypi() {
set -ex
pushd .
local mxnet_variant=${1:?"This function requires a python command as the
first argument"}
- if [[ ${mxnet_variant} = aarch64_cpu ]]; then
- source /opt/rh/rh-python38/enable
- fi
./cd/python/pypi/pypi_package.sh ${mxnet_variant}
popd
}
@@ -2044,10 +2026,6 @@ cd_integration_test_pypi() {
test_conv_params="--gpu"
fi
- if [[ ${mxnet_variant} = aarch64_cpu ]]; then
- source /opt/rh/rh-python38/enable
- fi
-
# install mxnet wheel package
python3 -m pip install --user ./wheel_build/dist/*.whl
diff --git a/ci/docker_cache.py b/ci/docker_cache.py
index a51203480d..f349a804b2 100755
--- a/ci/docker_cache.py
+++ b/ci/docker_cache.py
@@ -81,11 +81,14 @@ def _build_save_container(platform, registry, load_cache)
-> Optional[str]:
if build_util.is_docker_compose(platform):
if "dkr.ecr" in registry:
_ecr_login(registry)
- build_util.build_docker(platform=platform, registry=registry,
- num_retries=DOCKER_BUILD_NUM_RETRIES,
no_cache=False)
docker_compose_service = platform.split(".")[1]
env = os.environ.copy()
env["DOCKER_CACHE_REGISTRY"] = registry
+ if load_cache:
+ push_cmd = ['docker-compose', '-f', 'docker/docker-compose.yml',
'pull', '--quiet', docker_compose_service]
+ subprocess.check_call(push_cmd, env=env)
+ build_util.build_docker(platform=platform, registry=registry,
+ num_retries=DOCKER_BUILD_NUM_RETRIES,
no_cache=False)
push_cmd = ['docker-compose', '-f', 'docker/docker-compose.yml',
'push', docker_compose_service]
subprocess.check_call(push_cmd, env=env)
return None
diff --git a/ci/jenkins/Jenkins_steps.groovy b/ci/jenkins/Jenkins_steps.groovy
index 42e52a3c17..c80b71d2b2 100644
--- a/ci/jenkins/Jenkins_steps.groovy
+++ b/ci/jenkins/Jenkins_steps.groovy
@@ -147,7 +147,7 @@ def compile_unix_int64_gpu() {
ws('workspace/build-gpu-int64') {
timeout(time: max_time, unit: 'MINUTES') {
utils.init_git()
- utils.docker_run('ubuntu_gpu_cu101',
'build_ubuntu_gpu_large_tensor', false)
+ utils.docker_run('ubuntu_gpu_cu117',
'build_ubuntu_gpu_large_tensor', false)
utils.pack_lib('ubuntu_gpu_int64', mx_cmake_lib)
}
}
@@ -203,7 +203,7 @@ def compile_unix_mkldnn_gpu() {
ws('workspace/build-mkldnn-gpu') {
timeout(time: max_time, unit: 'MINUTES') {
utils.init_git()
- utils.docker_run('ubuntu_build_cuda', 'build_ubuntu_gpu_mkldnn',
false)
+ utils.docker_run('ubuntu_gpu_cu117', 'build_ubuntu_gpu_mkldnn',
false)
utils.pack_lib('mkldnn_gpu', mx_mkldnn_lib)
}
}
@@ -217,7 +217,7 @@ def compile_unix_mkldnn_nocudnn_gpu() {
ws('workspace/build-mkldnn-gpu-nocudnn') {
timeout(time: max_time, unit: 'MINUTES') {
utils.init_git()
- utils.docker_run('ubuntu_build_cuda',
'build_ubuntu_gpu_mkldnn_nocudnn', false)
+ utils.docker_run('ubuntu_gpu_cu117',
'build_ubuntu_gpu_mkldnn_nocudnn', false)
utils.pack_lib('mkldnn_gpu_nocudnn', mx_mkldnn_lib)
}
}
@@ -226,12 +226,12 @@ def compile_unix_mkldnn_nocudnn_gpu() {
}
def compile_unix_full_gpu() {
- return ['GPU: CUDA10.1+cuDNN7': {
+ return ['GPU: CUDA11.7+cuDNN8': {
node(NODE_LINUX_CPU) {
ws('workspace/build-gpu') {
timeout(time: max_time, unit: 'MINUTES') {
utils.init_git()
- utils.docker_run('ubuntu_build_cuda',
'build_ubuntu_gpu_cuda101_cudnn7', false)
+ utils.docker_run('ubuntu_gpu_cu117', 'build_ubuntu_gpu_cudnn8',
false)
utils.pack_lib('gpu', mx_lib_cpp_examples)
}
}
@@ -239,27 +239,13 @@ def compile_unix_full_gpu() {
}]
}
-def compile_unix_full_gpu_cu110() {
- return ['GPU: CUDA11.0+cuDNN8': {
- node(NODE_LINUX_CPU) {
- ws('workspace/build-gpu') {
- timeout(time: max_time, unit: 'MINUTES') {
- utils.init_git()
- utils.docker_run('ubuntu_build_cuda110',
'build_ubuntu_gpu_cuda110_cudnn8', false)
- utils.pack_lib('gpu_cu110', mx_lib_cpp_examples)
- }
- }
- }
- }]
-}
-
def compile_unix_full_gpu_mkldnn_cpp_test() {
- return ['GPU: CUDA10.1+cuDNN7+MKLDNN+CPPTEST': {
+ return ['GPU: CUDA11.7+cuDNN8+MKLDNN+CPPTEST': {
node(NODE_LINUX_CPU) {
ws('workspace/build-gpu-mkldnn-cpp') {
timeout(time: max_time, unit: 'MINUTES') {
utils.init_git()
- utils.docker_run('ubuntu_build_cuda',
'build_ubuntu_gpu_cuda101_cudnn7_mkldnn_cpp_test', false)
+ utils.docker_run('ubuntu_gpu_cu117',
'build_ubuntu_gpu_cudnn8_mkldnn_cpp_test', false)
utils.pack_lib('gpu_mkldnn_cpp_test', mx_lib_cpp_capi)
}
}
@@ -273,7 +259,7 @@ def compile_unix_cmake_mkldnn_gpu() {
ws('workspace/build-cmake-mkldnn-gpu') {
timeout(time: max_time, unit: 'MINUTES') {
utils.init_git()
- utils.docker_run('ubuntu_gpu_cu101',
'build_ubuntu_gpu_cmake_mkldnn', false)
+ utils.docker_run('ubuntu_gpu_cu117',
'build_ubuntu_gpu_cmake_mkldnn', false)
utils.pack_lib('cmake_mkldnn_gpu', mx_cmake_mkldnn_lib)
}
}
@@ -287,7 +273,7 @@ def compile_unix_cmake_gpu() {
ws('workspace/build-cmake-gpu') {
timeout(time: max_time, unit: 'MINUTES') {
utils.init_git()
- utils.docker_run('ubuntu_gpu_cu101', 'build_ubuntu_gpu_cmake',
false)
+ utils.docker_run('ubuntu_gpu_cu117', 'build_ubuntu_gpu_cmake',
false)
utils.pack_lib('cmake_gpu', mx_cmake_lib_cython)
}
}
@@ -301,7 +287,7 @@ def compile_unix_cmake_gpu_no_rtc() {
ws('workspace/build-cmake-gpu-no-rtc') {
timeout(time: max_time, unit: 'MINUTES') {
utils.init_git()
- utils.docker_run('ubuntu_gpu_cu101',
'build_ubuntu_gpu_cmake_no_rtc', false)
+ utils.docker_run('ubuntu_gpu_cu117',
'build_ubuntu_gpu_cmake_no_rtc', false)
}
}
}
@@ -356,7 +342,7 @@ def compile_centos7_gpu() {
ws('workspace/build-centos7-gpu') {
timeout(time: max_time, unit: 'MINUTES') {
utils.init_git()
- utils.docker_run('centos7_gpu_cu101', 'build_centos7_gpu', false)
+ utils.docker_run('centos7_gpu_cu117', 'build_centos7_gpu', false)
utils.pack_lib('centos7_gpu', mx_lib)
}
}
@@ -642,7 +628,7 @@ def test_static_scala_cpu() {
ws('workspace/ut-publish-scala-cpu') {
timeout(time: max_time, unit: 'MINUTES') {
utils.init_git()
- utils.docker_run('publish.ubuntu1404_cpu',
'build_static_scala_cpu', false)
+ utils.docker_run('centos7_cd', 'build_static_scala_cpu', false)
}
}
}
@@ -730,19 +716,7 @@ def test_unix_python3_gpu() {
node(NODE_LINUX_GPU_G4) {
ws('workspace/ut-python3-gpu') {
utils.unpack_and_init('gpu', mx_lib_cython)
- python3_gpu_ut_cython('ubuntu_gpu_cu101')
- utils.publish_test_coverage()
- }
- }
- }]
-}
-
-def test_unix_python3_gpu_cu110() {
- return ['Python3+CUDA11.0: GPU': {
- node(NODE_LINUX_GPU_G4) {
- ws('workspace/ut-python3-gpu') {
- utils.unpack_and_init('gpu_cu110', mx_lib_cython)
- python3_gpu_ut_cython('ubuntu_gpu_cu110')
+ python3_gpu_ut_cython('ubuntu_gpu_cu117')
utils.publish_test_coverage()
}
}
@@ -755,21 +729,7 @@ def test_unix_python3_quantize_gpu() {
ws('workspace/ut-python3-quantize-gpu') {
timeout(time: max_time, unit: 'MINUTES') {
utils.unpack_and_init('gpu', mx_lib)
- utils.docker_run('ubuntu_gpu_cu101',
'unittest_ubuntu_python3_quantization_gpu', true)
- utils.publish_test_coverage()
- }
- }
- }
- }]
-}
-
-def test_unix_python3_quantize_gpu_cu110() {
- return ['Python3+CUDA11.0: Quantize GPU': {
- node(NODE_LINUX_GPU_G4) {
- ws('workspace/ut-python3-quantize-gpu') {
- timeout(time: max_time, unit: 'MINUTES') {
- utils.unpack_and_init('gpu_cu110', mx_lib)
- utils.docker_run('ubuntu_gpu_cu110',
'unittest_ubuntu_python3_quantization_gpu', true)
+ utils.docker_run('ubuntu_gpu_cu117',
'unittest_ubuntu_python3_quantization_gpu', true)
utils.publish_test_coverage()
}
}
@@ -828,7 +788,7 @@ def test_unix_python3_mkldnn_gpu() {
node(NODE_LINUX_GPU_G4) {
ws('workspace/ut-python3-mkldnn-gpu') {
utils.unpack_and_init('mkldnn_gpu', mx_mkldnn_lib)
- python3_gpu_ut('ubuntu_gpu_cu101')
+ python3_gpu_ut('ubuntu_gpu_cu117')
utils.publish_test_coverage()
}
}
@@ -840,7 +800,7 @@ def test_unix_python3_mkldnn_nocudnn_gpu() {
node(NODE_LINUX_GPU_G4) {
ws('workspace/ut-python3-mkldnn-gpu-nocudnn') {
utils.unpack_and_init('mkldnn_gpu_nocudnn', mx_mkldnn_lib)
- python3_gpu_ut_nocudnn('ubuntu_gpu_cu101')
+ python3_gpu_ut_nocudnn('ubuntu_gpu_cu117')
utils.publish_test_coverage()
}
}
@@ -867,7 +827,7 @@ def test_unix_python3_integration_gpu() {
ws('workspace/it-python-gpu') {
timeout(time: max_time, unit: 'MINUTES') {
utils.unpack_and_init('gpu', mx_lib)
- utils.docker_run('ubuntu_gpu_cu101',
'integrationtest_ubuntu_gpu_python', true)
+ utils.docker_run('ubuntu_gpu_cu117',
'integrationtest_ubuntu_gpu_python', true)
utils.publish_test_coverage()
}
}
@@ -882,7 +842,7 @@ def test_unix_caffe_gpu() {
timeout(time: max_time, unit: 'MINUTES') {
utils.init_git()
utils.unpack_lib('gpu', mx_lib)
- utils.docker_run('ubuntu_gpu_cu101',
'integrationtest_ubuntu_gpu_caffe', true)
+ utils.docker_run('ubuntu_gpu_cu117',
'integrationtest_ubuntu_gpu_caffe', true)
utils.publish_test_coverage()
}
}
@@ -896,7 +856,7 @@ def test_unix_cpp_package_gpu() {
ws('workspace/it-cpp-package') {
timeout(time: max_time, unit: 'MINUTES') {
utils.unpack_and_init('gpu', mx_lib_cpp_examples)
- utils.docker_run('ubuntu_gpu_cu101',
'integrationtest_ubuntu_gpu_cpp_package', true)
+ utils.docker_run('ubuntu_gpu_cu117',
'integrationtest_ubuntu_gpu_cpp_package', true)
utils.publish_test_coverage()
}
}
@@ -910,7 +870,7 @@ def test_unix_capi_cpp_package() {
ws('workspace/it-capi-cpp-package') {
timeout(time: max_time, unit: 'MINUTES') {
utils.unpack_and_init('gpu_mkldnn_cpp_test', mx_lib_cpp_capi)
- utils.docker_run('ubuntu_gpu_cu101',
'integrationtest_ubuntu_gpu_capi_cpp_package', true)
+ utils.docker_run('ubuntu_gpu_cu117',
'integrationtest_ubuntu_gpu_capi_cpp_package', true)
utils.publish_test_coverage()
}
}
@@ -952,7 +912,7 @@ def test_unix_scala_gpu() {
ws('workspace/ut-scala-gpu') {
timeout(time: max_time, unit: 'MINUTES') {
utils.unpack_and_init('gpu', mx_lib)
- utils.docker_run('ubuntu_gpu_cu101',
'integrationtest_ubuntu_gpu_scala', true)
+ utils.docker_run('ubuntu_gpu_cu117',
'integrationtest_ubuntu_gpu_scala', true)
utils.publish_test_coverage()
}
}
@@ -1035,7 +995,7 @@ def test_unix_cpp_gpu() {
ws('workspace/ut-cpp-gpu') {
timeout(time: max_time, unit: 'MINUTES') {
utils.unpack_and_init('cmake_gpu', mx_cmake_lib)
- utils.docker_run('ubuntu_gpu_cu101', 'unittest_cpp', true)
+ utils.docker_run('ubuntu_gpu_cu117', 'unittest_cpp', true)
utils.publish_test_coverage()
}
}
@@ -1049,7 +1009,7 @@ def test_unix_cpp_mkldnn_gpu() {
ws('workspace/ut-cpp-mkldnn-gpu') {
timeout(time: max_time, unit: 'MINUTES') {
utils.unpack_and_init('cmake_mkldnn_gpu', mx_cmake_mkldnn_lib)
- utils.docker_run('ubuntu_gpu_cu101', 'unittest_cpp', true)
+ utils.docker_run('ubuntu_gpu_cu117', 'unittest_cpp', true)
utils.publish_test_coverage()
}
}
@@ -1077,7 +1037,7 @@ def test_unix_perl_gpu() {
ws('workspace/ut-perl-gpu') {
timeout(time: max_time, unit: 'MINUTES') {
utils.unpack_and_init('gpu', mx_lib)
- utils.docker_run('ubuntu_gpu_cu101',
'unittest_ubuntu_cpugpu_perl', true)
+ utils.docker_run('ubuntu_gpu_cu117',
'unittest_ubuntu_cpugpu_perl', true)
utils.publish_test_coverage()
}
}
@@ -1091,7 +1051,7 @@ def test_unix_r_gpu() {
ws('workspace/ut-r-gpu') {
timeout(time: max_time, unit: 'MINUTES') {
utils.unpack_and_init('gpu', mx_lib)
- utils.docker_run('ubuntu_gpu_cu101', 'unittest_ubuntu_gpu_R', true)
+ utils.docker_run('ubuntu_gpu_cu117', 'unittest_ubuntu_gpu_R', true)
utils.publish_test_coverage()
}
}
@@ -1159,7 +1119,7 @@ def test_unix_distributed_kvstore_gpu() {
ws('workspace/it-dist-kvstore') {
timeout(time: max_time, unit: 'MINUTES') {
utils.unpack_and_init('gpu', mx_lib)
- utils.docker_run('ubuntu_gpu_cu101',
'integrationtest_ubuntu_gpu_dist_kvstore', true)
+ utils.docker_run('ubuntu_gpu_cu117',
'integrationtest_ubuntu_gpu_dist_kvstore', true)
utils.publish_test_coverage()
}
}
@@ -1187,7 +1147,7 @@ def test_centos7_python3_gpu() {
ws('workspace/build-centos7-gpu') {
timeout(time: max_time, unit: 'MINUTES') {
utils.unpack_and_init('centos7_gpu', mx_lib)
- utils.docker_run('centos7_gpu_cu101', 'unittest_centos7_gpu', true)
+ utils.docker_run('centos7_gpu_cu117', 'unittest_centos7_gpu', true)
utils.publish_test_coverage()
}
}
@@ -1403,7 +1363,7 @@ def docs_julia() {
ws('workspace/docs') {
timeout(time: max_time, unit: 'MINUTES') {
utils.unpack_and_init('libmxnet', mx_lib, false)
- utils.docker_run('ubuntu_cpu_julia', 'build_julia_docs', false)
+ utils.docker_run('ubuntu_cpu', 'build_julia_docs', false)
if (should_pack_website()) {
utils.pack_lib('julia-artifacts',
'docs/_build/julia-artifacts.tgz', false)
}
diff --git a/ci/jenkins/Jenkinsfile_unix_gpu_cu110
b/ci/jenkins/Jenkinsfile_unix_gpu_cu110
deleted file mode 100644
index 3f663dc996..0000000000
--- a/ci/jenkins/Jenkinsfile_unix_gpu_cu110
+++ /dev/null
@@ -1,52 +0,0 @@
-// -*- mode: groovy -*-
-
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-//
-// Jenkins pipeline
-// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/
-
-// timeout in minutes
-max_time = 180
-
-node('utility') {
- // Loading the utilities requires a node context unfortunately
- checkout scm
- utils = load('ci/Jenkinsfile_utils.groovy')
- custom_steps = load('ci/jenkins/Jenkins_steps.groovy')
-}
-utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu',
linux_gpu: 'mxnetlinux-gpu', linux_gpu_p3: 'mxnetlinux-gpu-p3', linux_gpu_g4:
'mxnetlinux-gpu-g4')
-
-utils.main_wrapper(
-core_logic: {
- utils.parallel_stage('Build', [
- custom_steps.compile_unix_full_gpu_cu110()
- ])
-
- utils.parallel_stage('Tests', [
- custom_steps.test_unix_python3_gpu_cu110(),
- custom_steps.test_unix_python3_quantize_gpu_cu110()
- ])
-}
-,
-failure_handler: {
- // Only send email if master or release branches failed
- if (currentBuild.result == "FAILURE" && (env.BRANCH_NAME == "master" ||
env.BRANCH_NAME.startsWith("v"))) {
- emailext body: 'Build for MXNet branch ${BRANCH_NAME} has broken. Please
view the build at ${BUILD_URL}', replyTo: '${EMAIL}', subject: '[BUILD FAILED]
Branch ${BRANCH_NAME} build ${BUILD_NUMBER}', to: '${EMAIL}'
- }
-}
-)
diff --git a/ci/publish/Jenkinsfile b/ci/publish/Jenkinsfile
deleted file mode 100644
index ed09b4c2ef..0000000000
--- a/ci/publish/Jenkinsfile
+++ /dev/null
@@ -1,110 +0,0 @@
-// -*- mode: groovy -*-
-
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-// Jenkins pipeline
-// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/
-
-//mxnet libraries
-mx_scala_pub = 'lib/**, 3rdparty/dmlc-core/libdmlc.a,
3rdparty/tvm/nnvm/lib/libnnvm.a, 3rdparty/ps-lite/build/libps.a,
deps/lib/libprotobuf-lite.a, deps/lib/libzmq.a, config.mk,
scala-package/pom.xml, scala-package/**/pom.xml, scala-package/**/target/**,
scala-package/*/target/repo/**'
-
-// timeout in minutes
-max_time = 120
-
-node('restricted-utility') {
- // Loading the utilities requires a node context unfortunately
- checkout scm
- utils = load('ci/Jenkinsfile_utils.groovy')
-}
-utils.assign_node_labels(utility: 'restricted-utility', linux_cpu:
'restricted-mxnetlinux-cpu', linux_gpu: 'restricted-mxnetlinux-gpu',
linux_gpu_p3: 'restricted-mxnetlinux-gpu-p3', windows_cpu:
'restricted-mxnetwindows-cpu', windows_gpu: 'restricted-mxnetwindows-gpu')
-
-// CPU and GPU. OSX nodes are not currently supported by Jenkins
-def nodeMap = ['cpu': NODE_LINUX_CPU, 'gpu': NODE_LINUX_GPU_P3]
-def scalaOSMap = ['cpu': 'linux-x86_64-cpu', 'gpu': 'linux-x86_64-gpu']
-def scalaVariantMap = ['cpu': 'cpu', 'gpu': 'cu92']
-
-def wrapStep(nodeToRun, workspaceName, step) {
- return {
- node(nodeToRun) {
- ws("workspace/${workspaceName}") {
- timeout(time: max_time, unit: 'MINUTES') {
- step()
- }
- }
- }
- }
-}
-
-def toBuild = [:]
-def labels = ['cpu', 'gpu']
-for (x in labels) {
- def label = x // Required due to language
- toBuild["Scala Build ${label}"] = wrapStep(nodeMap['cpu'],
"build-scala-${label}") {
- withEnv(["MAVEN_PUBLISH_OS_TYPE=${scalaOSMap[label]}",
"mxnet_variant=${scalaVariantMap[label]}"]) {
- utils.init_git()
- utils.docker_run("publish.ubuntu1404_cpu", 'publish_scala_build', false,
'500m', 'MAVEN_PUBLISH_OS_TYPE mxnet_variant')
- utils.pack_lib("scala_${label}", mx_scala_pub, false)
- }
- }
-}
-
-def toTest = [:]
-def systems = ['ubuntu1604', 'ubuntu1804', 'centos7']
-for (x in labels) {
- def label = x // Required due to language
- for (y in systems) {
- def system = y // Required due to language
- toTest["Scala Test ${system} ${label}"] = wrapStep(nodeMap[label],
"test-scala-${system}-${label}") {
- withEnv(["mxnet_variant=${scalaVariantMap[label]}"]) {
- utils.unpack_and_init("scala_${label}", mx_scala_pub, false)
- utils.docker_run("publish.test.${system}_${label}",
'publish_scala_test', label == 'gpu', '500m', 'mxnet_variant')
- }
- }
- }
-}
-
-def toDeploy = [:]
-for (x in labels) {
- def label = x // Required due to language
- toDeploy["Scala Deploy ${label}"] = wrapStep(nodeMap[label],
"deploy-scala-${label}") {
- withEnv(["MAVEN_PUBLISH_OS_TYPE=${scalaOSMap[label]}",
"mxnet_variant=${scalaVariantMap[label]}"]) {
- utils.unpack_and_init("scala_${label}", mx_scala_pub, false)
- utils.docker_run("publish.ubuntu1604_${label}", 'publish_scala_deploy',
label == 'gpu' ? true : false, '500m', 'MAVEN_PUBLISH_OS_TYPE
MAVEN_PUBLISH_SECRET_ENDPOINT_URL MAVEN_PUBLISH_SECRET_NAME_CREDENTIALS
MAVEN_PUBLISH_SECRET_NAME_GPG DOCKERHUB_SECRET_ENDPOINT_REGION mxnet_variant')
- }
- }
-}
-
-utils.main_wrapper(
-core_logic: {
- stage('Build Packages') {
- parallel toBuild
- }
- stage('Test Packages') {
- parallel toTest
- }
- stage('Deploy Packages') {
- parallel toDeploy
- }
-}
-,
-failure_handler: {
- if (currentBuild.result == "FAILURE") {
- emailext body: 'Generating the nightly maven has failed. Please view the
build at ${BUILD_URL}', replyTo: '${EMAIL}', subject: '[NIGHTLY MAVEN FAILED]
Build ${BUILD_NUMBER}', to: '${EMAIL}'
- }
-}
-)
diff --git a/python/mxnet/contrib/text/utils.py
b/python/mxnet/contrib/text/utils.py
index 79cf903edb..2250979754 100644
--- a/python/mxnet/contrib/text/utils.py
+++ b/python/mxnet/contrib/text/utils.py
@@ -77,7 +77,7 @@ def count_tokens_from_str(source_str, token_delim=' ',
seq_delim='\n',
source_str = [t.lower() for t in source_str]
if counter_to_update is None:
- return collections.Counter(source_str)
+ return collections.Counter(source_str) # pylint:
disable=too-many-function-args
else:
counter_to_update.update(source_str)
return counter_to_update
diff --git a/python/mxnet/numpy/multiarray.py b/python/mxnet/numpy/multiarray.py
index 9a803d48b5..7c5ef0092a 100644
--- a/python/mxnet/numpy/multiarray.py
+++ b/python/mxnet/numpy/multiarray.py
@@ -193,8 +193,8 @@ _NUMPY_ARRAY_FUNCTION_DICT = {}
_NUMPY_ARRAY_UFUNC_DICT = {}
-@set_module('mxnet.numpy') # pylint: disable=invalid-name
-class ndarray(NDArray):
+@set_module('mxnet.numpy')
+class ndarray(NDArray): # pylint: disable=invalid-name
"""
ndarray(handle, writable=True):
diff --git a/python/mxnet/optimizer/optimizer.py
b/python/mxnet/optimizer/optimizer.py
index 737a947509..c92c35e98e 100755
--- a/python/mxnet/optimizer/optimizer.py
+++ b/python/mxnet/optimizer/optimizer.py
@@ -1537,8 +1537,8 @@ class SGLD(Optimizer):
-@register # pylint: disable=invalid-name
-class ccSGD(SGD):
+@register
+class ccSGD(SGD): # pylint: disable=invalid-name
"""[DEPRECATED] Same as `SGD`. Left here for backward compatibility."""
def __init__(self, *args, **kwargs):
super(ccSGD, self).__init__(*args, **kwargs)
diff --git a/tests/nightly/Jenkinsfile b/tests/nightly/Jenkinsfile
index 192f3d3452..0d1c881fad 100755
--- a/tests/nightly/Jenkinsfile
+++ b/tests/nightly/Jenkinsfile
@@ -38,34 +38,6 @@ core_logic: {
}
}
},
- 'InstallationGuide: CPU': {
- node(NODE_LINUX_CPU) {
- ws('workspace/nt-Installation-cpu') {
- utils.init_git()
- //Some install guide tests are currently diabled and tracked here:
- //1. https://github.com/apache/incubator-mxnet/issues/11369
- //2. https://github.com/apache/incubator-mxnet/issues/11288
- //utils.docker_run('ubuntu_base_cpu', 'nightly_test_installation
ubuntu_python_cpu_virtualenv', false)
- //docker_run('ubuntu_base_cpu', 'nightly_test_installation
ubuntu_python_cpu_pip', false)
- //docker_run('ubuntu_base_cpu', 'nightly_test_installation
ubuntu_python_cpu_docker', false)
- //docker_run('ubuntu_base_cpu', 'nightly_test_installation
ubuntu_python_cpu_source', false)
- }
- }
- },
- 'InstallationGuide: GPU': {
- node(NODE_LINUX_GPU) {
- ws('workspace/nt-Installation-gpu') {
- utils.init_git()
- //Some install guide tests are currently diabled and tracked here:
- //1. https://github.com/apache/incubator-mxnet/issues/11369
- //2. https://github.com/apache/incubator-mxnet/issues/11288
- //utils.docker_run('ubuntu_base_gpu', 'nightly_test_installation
ubuntu_python_gpu_virtualenv', true)
- //docker_run('ubuntu_base_gpu', 'nightly_test_installation
ubuntu_python_gpu_pip', true)
- //docker_run('ubuntu_base_gpu', 'nightly_test_installation
ubuntu_python_gpu_docker', true)
- //utils.docker_run('ubuntu_base_gpu', 'nightly_test_installation
ubuntu_python_gpu_source', true)
- }
- }
- },
'PipTest: GPU': {
node(NODE_LINUX_GPU) {
ws('workspace/nt-pipTest') {
diff --git a/tests/nightly/JenkinsfileForBinaries
b/tests/nightly/JenkinsfileForBinaries
index e44ad415cf..c8f52c2b05 100755
--- a/tests/nightly/JenkinsfileForBinaries
+++ b/tests/nightly/JenkinsfileForBinaries
@@ -32,11 +32,11 @@ utils.assign_node_labels(utility: 'utility', linux_cpu:
'mxnetlinux-cpu', linux_
utils.main_wrapper(
core_logic: {
stage('Build') {
- parallel 'GPU: CUDA10.1+cuDNN7': {
+ parallel 'GPU: CUDA11.7+cuDNN8': {
node(NODE_LINUX_CPU) {
ws('workspace/build-mkldnn-gpu') {
utils.init_git()
- utils.docker_run('ubuntu_build_cuda', 'build_ubuntu_gpu_mkldnn',
false)
+ utils.docker_run('ubuntu_gpu_cu117', 'build_ubuntu_gpu_mkldnn',
false)
utils.pack_lib('gpu', mx_lib_cpp_example_mkl)
}
}
