This is an automated email from the ASF dual-hosted git repository.
marcoabreu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new f033aec Static build for Python (#13916)
f033aec is described below
commit f033aec603c75547f6abcb95d779650a0e1b5025
Author: Lanking <[email protected]>
AuthorDate: Wed Jan 23 14:11:23 2019 -0800
Static build for Python (#13916)
* add python unit test
* address comments
* switch sanity test to Gluon module test
* We don't run tests (╯‵□′)╯︵┻━┻
* add variant in the environment variable
* add document improvement
* kill the conflict
---
ci/docker/runtime_functions.sh | 8 ++++++++
ci/jenkins/Jenkins_steps.groovy | 13 +++++++++++++
ci/jenkins/Jenkinsfile_unix_cpu | 1 +
ci/publish/README.md | 16 ++++++++--------
ci/publish/python/build.sh | 26 ++++++++++++++++++++++++++
tools/staticbuild/README.md | 10 +++++-----
tools/staticbuild/build_wheel.sh | 5 +----
7 files changed, 62 insertions(+), 17 deletions(-)
diff --git a/ci/docker/runtime_functions.sh b/ci/docker/runtime_functions.sh
index 9f9dd0c..f763137 100755
--- a/ci/docker/runtime_functions.sh
+++ b/ci/docker/runtime_functions.sh
@@ -1295,6 +1295,14 @@ build_scala_static_mkl() {
popd
}
+build_static_python_mkl() {
+ set -ex
+ pushd .
+ export mxnet_variant=mkl
+ ./ci/publish/python/build.sh
+ popd
+}
+
publish_scala_build() {
set -ex
pushd .
diff --git a/ci/jenkins/Jenkins_steps.groovy b/ci/jenkins/Jenkins_steps.groovy
index 984672c..f1703ba 100644
--- a/ci/jenkins/Jenkins_steps.groovy
+++ b/ci/jenkins/Jenkins_steps.groovy
@@ -534,6 +534,19 @@ def test_static_scala_cpu() {
}]
}
+def test_static_python_cpu() {
+ return ['Static build CPU 14.04 Python' : {
+ node(NODE_LINUX_CPU) {
+ ws('workspace/ut-publish-python-cpu') {
+ timeout(time: max_time, unit: 'MINUTES') {
+ utils.init_git()
+ utils.docker_run("publish.ubuntu1404_cpu",
'build_static_python_mkl', false)
+ }
+ }
+ }
+ }]
+}
+
def test_unix_python2_cpu() {
return ['Python2: CPU': {
node(NODE_LINUX_CPU) {
diff --git a/ci/jenkins/Jenkinsfile_unix_cpu b/ci/jenkins/Jenkinsfile_unix_cpu
index 9446348..234a65b 100644
--- a/ci/jenkins/Jenkinsfile_unix_cpu
+++ b/ci/jenkins/Jenkinsfile_unix_cpu
@@ -60,6 +60,7 @@ core_logic: {
custom_steps.test_unix_onnx_cpu(),
custom_steps.test_unix_cpp_cpu(),
custom_steps.test_static_scala_cpu(),
+ custom_steps.test_static_python_cpu(),
/* Disabled due to master build failure:
*
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/1221/pipeline/
* https://github.com/apache/incubator-mxnet/issues/11801
diff --git a/ci/publish/README.md b/ci/publish/README.md
index 2344cdf..f1ece6f 100644
--- a/ci/publish/README.md
+++ b/ci/publish/README.md
@@ -1,9 +1,9 @@
# MXNet Publish Settings
-This folder contains the configuration of restricted node on Jenkins for the
publish. It also contains a folder called `scala` that contains everything
required for scala publish. In this `README`, we would bring a brief
walkthrough of the Jenkins configuration as well as the usages of the scala
deployment files.
+This folder contains the configuration for restricted nodes on Jenkins for the
publishing MXNet artifacts. It also contains a folder called `scala` that
contains everything required for publishing to Maven. In this `README`, we
provide a brief walkthrough of the Jenkins configuration as well as the usage
of the Scala deployment files. Python publishing is TBD.
## Jenkins
-Currently, Jenkins contains three build stages, namely `Build Packages`, `Test
Packages` and `Deploy Packages`. During the `build package` stages, all
dependencies will be built and a Scala package would be created. In the second
stage, the package created from the previous stage would move to this stage to
specifically run the tests. In the final stage, the packages passed the test
would be deployed by the instances.
+Currently, Jenkins contains three build stages, namely `Build Packages`, `Test
Packages` and `Deploy Packages`. During the `build package` stages, all
dependencies are built and a Scala package are created. In the second stage,
the package created from the previous stage moves to this stage to specifically
run the tests. In the final stage, the packages that pass the tests are
deployed by the instances.
The job is scheduled to be triggered every 24 hours on a [restricted
instance](http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/restricted-publish-artifacts).
@@ -17,11 +17,11 @@ All packages are currently built in `Ubuntu 14.04`. All
Dockerfile used for publ
Apart from that, the script used to create the environment and publish are
available under `ci/docker/install`:
-- `ubuntu_publish.sh` install all required dependencies for Ubuntu 14.04 for
publishing
-- `ubuntu_base.sh` install minimum dependencies required to run the published
packages
+- `ubuntu_publish.sh` installs all required dependencies for Ubuntu 14.04 for
publishing
+- `ubuntu_base.sh` installs minimum dependencies required to run the published
packages
-## Scala publish
-Currently Scala publish on Linux is fully supported on jenkins. The `scala/`
folder contains all files needed to do the publish. Here is a breif
instroduction of the files:
+## Scala publishing
+Currently Scala publish on Linux is fully supported on Jenkins. The `scala/`
folder contains all files needed for publishing. Here is a brief introduction
of the files:
- `build.sh` Main executable files to build the backend as well as scala
package
- `buildkey.py` Main file used to extract password from the system and
configure the maven
@@ -29,5 +29,5 @@ Currently Scala publish on Linux is fully supported on
jenkins. The `scala/` fol
- `fullDeploy.sh` Used by CI to make full publish
- `test.sh` Make Scala test on CI
-## Python
-We plans to support Python build on Jenkins soon
\ No newline at end of file
+## Python publishing
+Python build support is TBD.
diff --git a/ci/publish/python/build.sh b/ci/publish/python/build.sh
new file mode 100755
index 0000000..6154989
--- /dev/null
+++ b/ci/publish/python/build.sh
@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -ex
+
+source tools/staticbuild/build.sh $mxnet_variant pip
+
+set -ex
+
+# Compile tests for discovery later
+source tools/staticbuild/build_wheel.sh
diff --git a/tools/staticbuild/README.md b/tools/staticbuild/README.md
index 7cfeb78..2def768 100644
--- a/tools/staticbuild/README.md
+++ b/tools/staticbuild/README.md
@@ -1,9 +1,9 @@
-# MXNet Static build
+# MXNet Static Build
-This folder contains the core script used to build the static library. This
README would bring you the information and usages of the script in here. Please
be aware, all of the scripts are designed to be run under the root folder.
+This folder contains the core script used to build the static library. This
README provides information on how to use the scripts in this folder. Please be
aware, all of the scripts are designed to be run under the root folder.
## `build.sh`
-This script is a wrapper around `build_lib.sh` aimed to simplify the usage of
it. It would automatically identify the system version, number of cores and all
environment variable settings. Here are the examples you can run this script:
+This script is a wrapper around `build_lib.sh. It simplifies the things by
automatically identifing the system version, number of cores, and all
environment variable settings. Here are examples you can run with this script:
```
tools/staticbuild/build.sh cu92 maven
@@ -17,7 +17,7 @@ This would build the mxnet package based on MKLDNN and and
pypi configuration se
As the result, users would have a complete static dependencies in
`/staticdeps` in the root folder as well as a static-linked `libmxnet.so` file
lives in `lib`. You can build your language binding by using the `libmxnet.so`.
## `build_lib.sh`
-This script would clone the most up-to-date master and build the MXNet backend
with static library. In order to run that, you should have prepare the the
following environment variable:
+This script clones the most up-to-date master and builds the MXNet backend
with a static library. In order to run the static library, you must set the the
following environment variables:
- `DEPS_PATH` Path to your static dependencies
- `STATIC_BUILD_TARGET` Either `pip` or `maven` as your publish platform
@@ -29,4 +29,4 @@ It is not recommended to run this file alone since there are
a bunch of variable
After running this script, you would have everything you need ready in the
`/lib` folder.
## `build_wheel.sh`
-This script is used to build the python package as well as running a sanity
test
\ No newline at end of file
+This script builds the python package. It also runs a sanity test.
\ No newline at end of file
diff --git a/tools/staticbuild/build_wheel.sh b/tools/staticbuild/build_wheel.sh
index a796341..9c1803b 100755
--- a/tools/staticbuild/build_wheel.sh
+++ b/tools/staticbuild/build_wheel.sh
@@ -18,14 +18,11 @@
# under the License.
# This script builds the wheel for binary distribution and performs sanity
check.
-
-cd mxnet-build
echo $(git rev-parse HEAD) >> python/mxnet/COMMIT_HASH
-cd -
+cd python/
# Make wheel for testing
python setup.py bdist_wheel
wheel_name=$(ls -t dist | head -n 1)
pip install -U --user --force-reinstall dist/$wheel_name
-python sanity_test.py
