[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r206345666
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,82 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+set -ex
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version --user
+}
+
+## Cuts the string and gives only the major version part.
+## eg : 12.3.0 ---> 12
+get_major_version() {
+major=$(echo $1 | cut -d. -f1)
+echo $major
+}
+
+## We read the current major version from libinfo.py file. And we extract the
major version from it.
+curr_mxnet_version=$(grep -w "__version__" python/mxnet/libinfo.py | grep -o
'".*"' | sed 's/"//g')
+curr_major_version=$(get_major_version $curr_mxnet_version)
Review comment:
Please validate the output to ensure we get a notification/error if somebody
breaks this by accident.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r206315214
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version
+}
+
+install_boto3(){
+ echo "Intalling boto3"
+ pip install boto3
+}
+
+echo `pwd`
+cd tests/nightly/model_backwards_compatibility_check
+echo `pwd`
+
+install_boto3
+
+## Fetch the latest release tags, filtering out 'rcs' and filtering out some
other irrelevant ones
+## This list is sorted in descending order chronologically. Keeping n = 5 for
a precautionary check.
+## Sample output for the below git tag command is : 1.2.0 utils 1.1.0 1.0.0
0.12.1
+previous_versions=($(git tag --sort=-creatordate | grep --invert-match rc |
head -n 5))
+count=0
+for version in ${previous_versions[*]}
+do
+ # We just need to train the previous two versions. This logic can be
changed later on as welll.
+ if [[ "$count" -gt 1 ]]
Review comment:
As stated in-person, we have to train everything under the same major
version.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r206312686
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+set -ex
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version --user
+}
+
+echo `pwd`
+cd tests/nightly/model_backwards_compatibility_check
+echo `pwd`
+
+## Fetch the latest release tags, filtering out 'rcs' and filtering out some
other irrelevant ones
+## This list is sorted in descending order chronologically.
+## Sample output for the below git tag command is : 1.2.0 utils 1.1.0 1.0.0
0.12.1
+## so from this sample we will pick up the top two : 1.2.0 and 1.1.0 and train
models on them
+## Now while performing inference the latest version could be 1.3.0, which
will help in validating models trained
+## on 1.1.0 and 1.2.0 by loading them on the latest version (1.3.0)
+## Over a period of time, the model repository will grow since with every new
release we
+## upload models trained on newer versions as well through this script
+previous_versions=($(git tag --sort=-creatordate | grep --invert-match rc))
+count=0
+for version in ${previous_versions[*]}
+do
+ # We just need to train the previous two versions. This logic can be
changed later on as welll.
+ if [[ "$count" -gt 1 ]]
+ then
+ echo "Successfully trained files for the previous two MXNet
release versions"
+ exit 0
+ fi
+
+ ## If MXNet major version starts with a number >=1. with a wildcard
match for the minor version numbers
+ ## Could have used a [[:digit:]]+. as well but it was not working as a
traditional regex in bash.
+ ## so had to resort to using [[:digit:]] [[:digit:]]* to indicate
multi-digit version regex match
+ if [[ $version = [[:digit:][[:digit:]]*.[[:digit:]].[[:digit:]] ]]
Review comment:
Would this also cover 1.12.0? It seems like you only added multi digit for
the major version.
Also, could you explain how this exactly determines whether it is a major
version match?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r206247863 ## File path: tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh ## @@ -0,0 +1,67 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +#Author: Piyush Ghai + Review comment: Missing set -ex This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r206247731 ## File path: tests/nightly/model_backwards_compatibility_check/model_backward_compat_checker.sh ## @@ -0,0 +1,28 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +#Author: Piyush Ghai + Review comment: Missing set -ex This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205389227
##
File path: tests/nightly/model_backwards_compatibility_check/JenkinsfileForMBCC
##
@@ -0,0 +1,86 @@
+// -*- mode: groovy -*-
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+//This is a Jenkinsfile for the model backwards compatibility checker. The
format and some functions have been picked up from the top-level Jenkinsfile.
+
+err = null
+
+def init_git() {
+ deleteDir()
+ retry(5) {
+try {
+ timeout(time: 15, unit: 'MINUTES') {
+checkout scm
+sh 'git submodule update --init --recursive'
+sh 'git clean -d -f'
+ }
+} catch (exc) {
+ deleteDir()
+ error "Failed to fetch source codes with ${exc}"
+ sleep 2
+}
+ }
+}
+
+def docker_run(platform, function_name, use_nvidia, shared_mem = '500m') {
+ def command = "ci/build.py --docker-registry ${env.DOCKER_CACHE_REGISTRY}
%USE_NVIDIA% --platform %PLATFORM% --shm-size %SHARED_MEM%
/work/runtime_functions.sh %FUNCTION_NAME%"
+ command = command.replaceAll('%USE_NVIDIA%', use_nvidia ? '--nvidiadocker' :
'')
+ command = command.replaceAll('%PLATFORM%', platform)
+ command = command.replaceAll('%FUNCTION_NAME%', function_name)
+ command = command.replaceAll('%SHARED_MEM%', shared_mem)
+
+ sh command
+}
+
+try {
+ stage('MBCC'){
+parallel 'ModelBackwardsCompat: CPU': {
+ node('mxnetlinux-cpu') {
+ws('workspace/modelBackwardsCompat') {
+
+init_git()
+// Train models on older versions
+docker_run('ubuntu_nightly_cpu',
'nightly_model_backwards_compat_train', false)
+// upload files to S3 here outside of the docker environment
+sh
/tests/nightly/model_backwards_compatibility_check/upload_models_to_s3.sh
Review comment:
I this this path is wrong because it is absolute. I'd double check here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205390143
##
File path: ci/docker/runtime_functions.sh
##
@@ -895,6 +895,25 @@ nightly_test_javascript() {
make -C /work/mxnet/amalgamation libmxnet_predict.js MIN=1
EMCC=/work/deps/emscripten/emcc
}
+#Tests Model backwards compatibility on MXNet
+nightly_model_backwards_compat_test() {
+set -ex
+export PYTHONPATH=./python/
+
./tests/nightly/model_backwards_compatibility_check/model_backward_compat_checker.sh
+}
+
+#Backfills S3 bucket with models trained on earlier versions of mxnet
+nightly_model_backwards_compat_train() {
+set -ex
+export PYTHONPATH=./python/
+VENV=mbcc_py2_venv
+virtualenv -p `which python2` $VENV
Review comment:
If you run inside a virtualenv, you won't have access to any installed
packages. I'd recommend not to use virtualenv here because you're running
inside docker anyways. This is also the reason for the CI error that states it
could not find boto3.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205389097
##
File path: tests/nightly/model_backwards_compatibility_check/JenkinsfileForMBCC
##
@@ -0,0 +1,86 @@
+// -*- mode: groovy -*-
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+//This is a Jenkinsfile for the model backwards compatibility checker. The
format and some functions have been picked up from the top-level Jenkinsfile.
+
+err = null
+
+def init_git() {
+ deleteDir()
+ retry(5) {
+try {
+ timeout(time: 15, unit: 'MINUTES') {
+checkout scm
+sh 'git submodule update --init --recursive'
+sh 'git clean -d -f'
+ }
+} catch (exc) {
+ deleteDir()
+ error "Failed to fetch source codes with ${exc}"
+ sleep 2
+}
+ }
+}
+
+def docker_run(platform, function_name, use_nvidia, shared_mem = '500m') {
+ def command = "ci/build.py --docker-registry ${env.DOCKER_CACHE_REGISTRY}
%USE_NVIDIA% --platform %PLATFORM% --shm-size %SHARED_MEM%
/work/runtime_functions.sh %FUNCTION_NAME%"
+ command = command.replaceAll('%USE_NVIDIA%', use_nvidia ? '--nvidiadocker' :
'')
+ command = command.replaceAll('%PLATFORM%', platform)
+ command = command.replaceAll('%FUNCTION_NAME%', function_name)
+ command = command.replaceAll('%SHARED_MEM%', shared_mem)
+
+ sh command
+}
+
+try {
+ stage('MBCC'){
+parallel 'ModelBackwardsCompat: CPU': {
+ node('mxnetlinux-cpu') {
+ws('workspace/modelBackwardsCompat') {
+
+init_git()
+// Train models on older versions
+docker_run('ubuntu_nightly_cpu',
'nightly_model_backwards_compat_train', false)
Review comment:
Could you put the training and the testing into two separate (non-parallel)
stages to increase clarity?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205389280
##
File path: tests/nightly/model_backwards_compatibility_check/JenkinsfileForMBCC
##
@@ -0,0 +1,86 @@
+// -*- mode: groovy -*-
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+//This is a Jenkinsfile for the model backwards compatibility checker. The
format and some functions have been picked up from the top-level Jenkinsfile.
+
+err = null
+
+def init_git() {
+ deleteDir()
+ retry(5) {
+try {
+ timeout(time: 15, unit: 'MINUTES') {
+checkout scm
+sh 'git submodule update --init --recursive'
+sh 'git clean -d -f'
+ }
+} catch (exc) {
+ deleteDir()
+ error "Failed to fetch source codes with ${exc}"
+ sleep 2
+}
+ }
+}
+
+def docker_run(platform, function_name, use_nvidia, shared_mem = '500m') {
+ def command = "ci/build.py --docker-registry ${env.DOCKER_CACHE_REGISTRY}
%USE_NVIDIA% --platform %PLATFORM% --shm-size %SHARED_MEM%
/work/runtime_functions.sh %FUNCTION_NAME%"
+ command = command.replaceAll('%USE_NVIDIA%', use_nvidia ? '--nvidiadocker' :
'')
+ command = command.replaceAll('%PLATFORM%', platform)
+ command = command.replaceAll('%FUNCTION_NAME%', function_name)
+ command = command.replaceAll('%SHARED_MEM%', shared_mem)
+
+ sh command
+}
+
+try {
+ stage('MBCC'){
+parallel 'ModelBackwardsCompat: CPU': {
+ node('mxnetlinux-cpu') {
+ws('workspace/modelBackwardsCompat') {
+
+init_git()
+// Train models on older versions
+docker_run('ubuntu_nightly_cpu',
'nightly_model_backwards_compat_train', false)
+// upload files to S3 here outside of the docker environment
+sh
/tests/nightly/model_backwards_compatibility_check/upload_models_to_s3.sh
+// Perform inference on these models
+docker_run('ubuntu_nightly_cpu',
'nightly_model_backwards_compat_test', false)
+
+ }
Review comment:
Please check the indentation
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205195997
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version
+}
+
+install_boto3(){
+ echo "Intalling boto3"
+ pip install boto3
+}
+
+echo `pwd`
+cd tests/nightly/model_backwards_compatibility_check
+echo `pwd`
+
+install_boto3
+
+## Fetch the latest release tags, filtering out 'rcs' and filtering out some
other irrelevant ones
+## This list is sorted in descending order chronologically. Keeping n = 5 for
a precautionary check.
+## Sample output for the below git tag command is : 1.2.0 utils 1.1.0 1.0.0
0.12.1
+previous_versions=($(git tag --sort=-creatordate | grep --invert-match rc |
head -n 5))
+count=0
+for version in ${previous_versions[*]}
+do
+ # We just need to train the previous two versions. This logic can be
changed later on as welll.
+ if [[ "$count" -gt 1 ]]
Review comment:
What would happen if we change the major version? Please elaborate what
would happen with multiple minor versions as of now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205195594
##
File path: ci/docker/runtime_functions.sh
##
@@ -854,6 +854,29 @@ nightly_test_javascript() {
make -C /work/mxnet/amalgamation libmxnet_predict.js MIN=1
EMCC=/work/deps/emscripten/emcc
}
+#Tests Model backwards compatibility on MXNet
+nightly_model_backwards_compat_test() {
+set -ex
+export PYTHONPATH=./python/
+pip install boto3
Review comment:
Oh right, even better!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r204630645
##
File path: ci/docker/runtime_functions.sh
##
@@ -854,6 +854,27 @@ nightly_test_javascript() {
make -C /work/mxnet/amalgamation libmxnet_predict.js MIN=1
EMCC=/work/deps/emscripten/emcc
}
+#Tests Model backwards compatibility on MXNet
+nightly_model_backwards_compat_test() {
+set -ex
+export PYTHONPATH=./python/
+pip install boto3
+
./tests/nightly/model_backwards_compatibility_check/model_backward_compat_checker.sh
+}
+
+#Backfills S3 bucket with models trained on earlier versions of mxnet
+nightly_model_backwards_compat_train() {
+set -ex
+export PYTHONPATH=./python/
+VENV=mbcc_py2_venv
+virtualenv -p `which python2` $VENV
+source $VENV/bin/activate
+pip install boto3
Review comment:
Sorry, retrospectively, my previous comment was not very clear. Please
excuse that.
Everything inside runtime_functions.sh is intended to be run within docker
**only**. Please have a look at
https://github.com/apache/incubator-mxnet/blob/master/tests/nightly/broken_link_checker_test/JenkinsfileForBLC#L57
to see how to combine the usage of Docker together with the integration of
boto to fulfil that constraint. It basically boils down to doing the core logic
inside the Docker environment and then running a second script (in this case
it's pure shell, but you can also create a second python script dedicated to
the publishing process). Feel free to hit me up if you need further advice here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r204631030
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version
+}
+
+install_boto3(){
Review comment:
As stated previously, please remove installing boto3 here
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r204630788
##
File path: ci/docker/runtime_functions.sh
##
@@ -854,6 +854,29 @@ nightly_test_javascript() {
make -C /work/mxnet/amalgamation libmxnet_predict.js MIN=1
EMCC=/work/deps/emscripten/emcc
}
+#Tests Model backwards compatibility on MXNet
+nightly_model_backwards_compat_test() {
+set -ex
+export PYTHONPATH=./python/
+pip install boto3
Review comment:
Please install boto here:
https://github.com/apache/incubator-mxnet/blob/master/ci/docker/install/ubuntu_core.sh
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r204629401
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version
+}
+
+install_boto3(){
+ echo "Intalling boto3"
+ pip install boto3
+}
+
+echo `pwd`
+cd tests/nightly/model_backwards_compatibility_check
+echo `pwd`
+
+install_boto3
+
+## Fetch the latest release tags, filtering out 'rcs' and filtering out some
other irrelevant ones
+## This list is sorted in descending order chronologically. Keeping n = 5 for
a precautionary check.
+## Sample output for the below git tag command is : 1.2.0 utils 1.1.0 1.0.0
0.12.1
+previous_versions=($(git tag --sort=-creatordate | grep --invert-match rc |
head -n 5))
+count=0
+for version in ${previous_versions[*]}
+do
+ # We just need to train the previous two versions. This logic can be
changed later on as welll.
+ if [[ "$count" -gt 1 ]]
Review comment:
I see, thank you. Wasn't it part of the initial design that we test
everything under the same major version, independent of the number of releases?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r203429780
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version
+}
+
+install_boto3(){
+ echo "Intalling boto3"
+ pip install boto3
+}
+
+echo `pwd`
+cd tests/nightly/model_backwards_compatibility_check
+echo `pwd`
+
+install_boto3
+
+## Fetch the latest release tags, filtering out 'rcs' and filtering out some
other irrelevant ones
+## This list is sorted in descending order chronologically. Keeping n = 5 for
a precautionary check.
+## Sample output for the below git tag command is : 1.2.0 utils 1.1.0 1.0.0
0.12.1
+previous_versions=($(git tag --sort=-creatordate | grep --invert-match rc |
head -n 5))
+count=0
+for version in ${previous_versions[*]}
+do
+ # We just need to train the previous two versions. This logic can be
changed later on as welll.
+ if [[ "$count" -gt 1 ]]
+ then
+ echo "Successfully trained files for the previous two MXNet
release versions"
+ exit 1
+ fi
+
+ ## If MXNet major version starts with a number >=1. with a wildcard
match for the minor version numbers
Review comment:
How does this handle different major versions?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r203428535 ## File path: tests/nightly/model_backwards_compatibility_check/model_backward_compat_checker.sh ## @@ -0,0 +1,28 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +#Author: Piyush Ghai + +echo "Invoking model_backwards_compat_checker.sh script" +echo `pwd` +cd tests/nightly/model_backwards_compatibility_check +echo `pwd` + +echo '==' +python model_backwards_compat_inference.py Review comment: Wouldn't this rather be in runtime_functions.sh? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r20342
##
File path: ci/docker/runtime_functions.sh
##
@@ -854,6 +854,27 @@ nightly_test_javascript() {
make -C /work/mxnet/amalgamation libmxnet_predict.js MIN=1
EMCC=/work/deps/emscripten/emcc
}
+#Tests Model backwards compatibility on MXNet
+nightly_model_backwards_compat_test() {
+set -ex
+export PYTHONPATH=./python/
+pip install boto3
+
./tests/nightly/model_backwards_compatibility_check/model_backward_compat_checker.sh
+}
+
+#Backfills S3 bucket with models trained on earlier versions of mxnet
+nightly_model_backwards_compat_train() {
+set -ex
+export PYTHONPATH=./python/
+VENV=mbcc_py2_venv
+virtualenv -p `which python2` $VENV
+source $VENV/bin/activate
+pip install boto3
Review comment:
It's still there, what did you change?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r203429160
##
File path:
tests/nightly/model_backwards_compatibility_check/model_backwards_compat_inference.py
##
@@ -0,0 +1,149 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from common import *
+
+def test_module_checkpoint_api():
+ model_name = 'module_checkpoint_api'
+ print ('Performing inference for model/API %s' %model_name)
+ data = download_data_from_s3(model_name)
+ if data is None:
+ print ('No data files found for %s' %model_name)
+ return
+
+ test_data = data['data']
+ test_label = data['labels']
+
+ data_iter = mx.io.NDArrayIter(test_data, test_label, batch_size=10)
+
+ ## For each MXNet version that has the saved models
+ for folder in get_top_level_folders_in_bucket(s3, model_bucket_name):
+ print ('Fetching files for MXNet version : %s and model %s'
%(folder, model_name))
+ model_files = download_model_files_from_s3(model_name, folder)
+ if len(model_files) == 0:
+ print('No training files found for %s for MXNet version
: %s'%(model_name, folder))
+ continue
+ ## Load the model and perform inference
+ loaded_model = get_module_api_model_definition()
+
+ sym, arg_params, aux_params =
mx.model.load_checkpoint(model_name, 1)
+ loaded_model.bind(data_shapes=data_iter.provide_data,
label_shapes=data_iter.provide_label)
+ loaded_model.set_params(arg_params, aux_params)
+
+ old_inference_results = load_inference_results(model_name)
+ inference_results = loaded_model.predict(data_iter)
+ ## Check whether they are equal or not ?
+ assert_almost_equal(inference_results.asnumpy(),
old_inference_results.asnumpy())
+ clean_model_files(model_files, model_name)
+ print ('=')
+
+ print ('Assertion passed for model : %s' %model_name)
+
+
+def test_lenet_gluon_load_params_api():
+ model_name = 'lenet_gluon_save_params_api'
+ print ('Performing inference for model/API %s' %model_name)
+ ## Get data from S3
+ data = download_data_from_s3(model_name)
+ if data is None:
+ print ('No data files found for %s' %model_name)
+ return
+
+ test_data = data['data']
+
+ for folder in get_top_level_folders_in_bucket(s3, model_bucket_name):
+ print ('Fetching files for MXNet version : %s and model %s'
%(folder, model_name))
+ model_files = download_model_files_from_s3(model_name, folder)
+ if len(model_files) == 0:
+ print('No training files found for %s for MXNet version
: %s'%(model_name, folder))
+ continue
+ ## Load the model and perform inference
+ loaded_model = Net()
+ loaded_model.load_params(model_name+'-params')
+ output = loaded_model(test_data)
+ old_inference_results = mx.nd.load(model_name +
'-inference')['inference']
+ assert_almost_equal(old_inference_results.asnumpy(),
output.asnumpy())
+ clean_model_files(model_files, model_name)
+ print ('=')
+ print ('Assertion passed for model : %s' %model_name)
+
+def test_lenet_gluon_hybrid_imports_api():
Review comment:
Do we really need to make a test case for each individual model? Wouldn't it
be possible to just loop through the S3 bucket and automatically determine all
models that have to be run? That way, we can add things as required without
copy this boilerplate.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r203429712
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version
+}
+
+install_boto3(){
+ echo "Intalling boto3"
+ pip install boto3
+}
+
+echo `pwd`
+cd tests/nightly/model_backwards_compatibility_check
+echo `pwd`
+
+install_boto3
+
+## Fetch the latest release tags, filtering out 'rcs' and filtering out some
other irrelevant ones
+## This list is sorted in descending order chronologically. Keeping n = 5 for
a precautionary check.
+## Sample output for the below git tag command is : 1.2.0 utils 1.1.0 1.0.0
0.12.1
+previous_versions=($(git tag --sort=-creatordate | grep --invert-match rc |
head -n 5))
+count=0
+for version in ${previous_versions[*]}
+do
+ # We just need to train the previous two versions. This logic can be
changed later on as welll.
+ if [[ "$count" -gt 1 ]]
Review comment:
Could you elaborate? We have to train everything that's missing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r203428140
##
File path: tests/nightly/model_backwards_compatibility_check/common.py
##
@@ -0,0 +1,210 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import boto3
+import mxnet as mx
+import json
+import os
+import numpy as np
+import logging
+from mxnet import nd, autograd, gluon
+import mxnet.ndarray as nd
+from mxnet.gluon.data.vision import transforms, datasets
+from mxnet import autograd as ag
+import mxnet.ndarray as F
+from mxnet.gluon import nn, rnn
+import re
+import time
+import sys
+from mxnet.test_utils import assert_almost_equal
+
+# Set fixed random seeds.
+mx.random.seed(7)
+np.random.seed(7)
+logging.getLogger().setLevel(logging.DEBUG)
+
+# get the current mxnet version we are running on
+mxnet_version = mx.__version__
+model_bucket_name = 'mxnet-model-backwards-compatibility-models'
+data_folder = 'mxnet-model-backwards-compatibility-data'
+backslash = '/'
+s3 = boto3.resource('s3')
+ctx = mx.cpu(0)
+
+def get_module_api_model_definition():
+input = mx.symbol.Variable('data')
+input = mx.symbol.Flatten(data=input)
+
+fc1 = mx.symbol.FullyConnected(data=input, name='fc1', num_hidden=128)
+act1 = mx.sym.Activation(data=fc1, name='relu1', act_type="relu")
+fc2 = mx.symbol.FullyConnected(data=fc1, name='fc2', num_hidden=2)
+op = mx.symbol.SoftmaxOutput(data=fc2, name='softmax')
+model = mx.mod.Module(symbol=op, context=ctx, data_names=['data'],
label_names=['softmax_label'])
+return model
+
+def save_inference_results(inference_results, model_name):
+assert (isinstance(inference_results, mx.ndarray.ndarray.NDArray))
+mx.nd.save(model_name + '-inference', {'inference' : inference_results})
+
+def load_inference_results(model_name):
+inf_dict = mx.nd.load(model_name+'-inference')
+return inf_dict['inference']
+
+def save_data_and_labels(test_data, test_labels, model_name):
+assert (isinstance(test_data, mx.ndarray.ndarray.NDArray))
+assert (isinstance(test_labels, mx.ndarray.ndarray.NDArray))
+mx.nd.save(model_name + '-data', {'data' : test_data, 'labels' :
test_labels})
+
+def upload_data_and_labels_to_s3(model_name):
+s3 = boto3.client('s3')
+file = model_name + '-data'
+s3.upload_file(file, model_bucket_name, data_folder + backslash + file)
+print ('data files successfully uploaded to s3')
+
+def upload_model_files_to_s3(files, folder_name):
+s3 = boto3.client('s3')
+for file in files:
+s3.upload_file(file, model_bucket_name, folder_name + file)
+
+def clean_model_files(files, model_name):
+files.append(model_name + '-inference')
+files.append(model_name + '-data')
+
+for file in files:
+if os.path.isfile(file):
+os.remove(file)
+
+def download_data_from_s3(model_name):
+print ('Downloading data files for %s from bucket %s'%(model_name,
model_bucket_name + backslash + data_folder))
+bucket = s3.Bucket(model_bucket_name)
+prefix = data_folder + backslash + model_name + '-data'
+data_files_meta = list(bucket.objects.filter(Prefix = prefix))
+if len(data_files_meta) == 0:
+print ('No data files found for %s' %model_name)
Review comment:
Please use logging instead of print
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r202282243
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version
+}
+
+install_boto3(){
+ echo "Intalling boto3"
+ pip install boto3
+}
+
+echo `pwd`
+cd tests/nightly/model_backwards_compatibility_check
+echo `pwd`
+
+install_boto3
+
+install_mxnet 1.1.0
Review comment:
Could we please make this list automatic by traversing the git branches?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r202282010
##
File path:
tests/nightly/model_backwards_compatibility_check/model_backwards_compat_train.py
##
@@ -0,0 +1,159 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from common import *
+
+def train_module_checkpoint_api():
+ model_name = 'module_checkpoint_api'
+ print ('Saving files for model %s' %model_name)
+ ### Prepare data
+ test_data = mx.nd.array(np.random.uniform(-1, 1, size=(20, 1)))
+ test_label = mx.nd.array(np.random.randint(0, 2, size=(20,)),
dtype='float32')
+ data_iter = mx.io.NDArrayIter(test_data, test_label, batch_size=10)
+
+
+ mod = get_module_api_model_definition()
+ mod.bind(data_shapes=data_iter.provide_data,
label_shapes=data_iter.provide_label)
+ weights = mx.initializer.Xavier(magnitude = 2.57)
+ mod.init_params(weights)
+
+ mod.save_checkpoint(model_name, 1)
+ ### Save the data, labels
+ save_data_and_labels(test_data, test_label, model_name)
+ upload_data_and_labels_to_s3(model_name)
+
+ inference_results = mod.predict(data_iter)
+ ### Save inference_results
+ save_inference_results(inference_results, model_name)
+ ### upload model and inference files to S3
+ files = list()
+ files.append(model_name + '-0001.params')
+ files.append(model_name + '-symbol.json')
+ files.append(model_name + '-inference')
+
+ mxnet_folder = str(mxnet_version) + backslash + model_name + backslash
+
+ upload_model_files_to_s3(files, mxnet_folder)
+
+ clean_model_files(files, model_name)
+
+def train_lenet_gluon_save_params_api():
+ model_name = 'lenet_gluon_save_params_api'
+ print ('Saving files for model %s' %model_name)
+ net = Net()
+ weights = mx.initializer.Xavier(magnitude = 2.57)
+ net.initialize(weights, ctx = [mx.cpu(0)])
+ ### Prepare data
+
+ test_data = mx.nd.array(np.random.uniform(-1, 1, size=(20, 1, 30, 30)))
+ output = net(test_data)
+ # print (y)
+ #### Save the test data as well.
+ #### Save the inference output ys
+ #### Save the model params
+
+ mx.nd.save(model_name + '-data', {'data' : test_data})
+ save_inference_results(output, model_name)
+ net.save_params(model_name + '-params')
+
+ mxnet_folder = str(mxnet_version) + backslash + model_name + backslash
+
+ files = list()
+ files.append(model_name + '-data')
+ files.append(model_name + '-inference')
+ files.append(model_name + '-params')
+
+ upload_data_and_labels_to_s3(model_name)
+
+ upload_model_files_to_s3(files, mxnet_folder)
+
+ clean_model_files(files, model_name)
+
+def train_lenet_gluon_hybrid_export_api():
+ model_name = 'lenet_gluon_hybrid_export_api'
+ print ('Saving files for model %s' %model_name)
+ net = HybridNet()
+ weights = mx.initializer.Xavier(magnitude = 2.57)
+ net.initialize(weights, ctx = [mx.cpu(0)])
+ net.hybridize()
+ ### Prepare data
+ test_data = mx.nd.array(np.random.uniform(-1, 1, size=(20, 1, 30, 30)))
+ output = net(test_data)
+ # print (y)
+### Save the test data as well.
+### Save the inference output ys
+### Save the model params
+
+ mx.nd.save(model_name + '-data', {'data' : test_data})
+ save_inference_results(output, model_name)
+ net.export(model_name, epoch=1)
+
+ mxnet_folder = str(mxnet_version) + backslash + model_name + backslash
+
+ files = list()
+ files.append(model_name + '-data')
+ files.append(model_name + '-inference')
+ files.append(model_name + '-0001.params')
+ files.append(model_name + '-symbol.json')
+
+
+ upload_data_and_labels_to_s3(model_name)
+
+ upload_model_files_to_s3(files, mxnet_folder)
+
+ clean_model_files(files, model_name)
+
+def train_lstm_gluon_save_parameters_api():
+ ## If this code is being run on version >= 1.2.0 only then execute it,
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r202281767
##
File path:
tests/nightly/model_backwards_compatibility_check/model_backwards_compat_inference.py
##
@@ -0,0 +1,168 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from common import *
+
+def test_module_checkpoint_api():
+ model_name = 'module_checkpoint_api'
+ print ('Performing inference for model/API %s' %model_name)
+ data = download_data_from_s3(model_name)
+ test_data = data['data']
+ test_label = data['labels']
+
+ data_iter = mx.io.NDArrayIter(test_data, test_label, batch_size=10)
+
+ ## For each MXNet version that has the saved models
+ for folder in get_top_level_folders_in_bucket(s3, model_bucket_name):
+ bucket = s3.Bucket(model_bucket_name)
+ prefix = folder + backslash + model_name
+ model_files_meta = list(bucket.objects.filter(Prefix = prefix))
+ if len(model_files_meta) == 0:
+ print ('No trained models found under path : %s' %prefix)
+ continue
+ model_files = list()
+ ## For each file found under a model folder :
+ for obj in model_files_meta:
+ file_name = obj.key.split('/')[2]
+ model_files.append(file_name)
+ ## Download this file---
+ bucket.download_file(obj.key, file_name)
+
+ ## Load the model and perform inference
+ loaded_model = get_module_api_model_definition()
+
+ sym, arg_params, aux_params =
mx.model.load_checkpoint(model_name, 1)
+ loaded_model.bind(data_shapes=data_iter.provide_data,
label_shapes=data_iter.provide_label)
+ loaded_model.set_params(arg_params, aux_params)
+
+ old_inference_results = load_inference_results(model_name)
+ inference_results = loaded_model.predict(data_iter)
+ ## Check whether they are equal or not ?
+ assert_almost_equal(inference_results.asnumpy(),
old_inference_results.asnumpy())
+ clean_model_files(model_files, model_name)
+
+ print ('Assertion passed for model : %s' %model_name)
+
+
+def test_lenet_gluon_load_params_api():
+ model_name = 'lenet_gluon_save_params_api'
+ print ('Performing inference for model/API %s' %model_name)
+ ## Get data from S3
+ data = download_data_from_s3(model_name)
+
+ test_data = data['data']
+
+ for folder in get_top_level_folders_in_bucket(s3, model_bucket_name):
Review comment:
This seems like a lot of duplicate code. Could we extract it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r202281614
##
File path:
tests/nightly/model_backwards_compatibility_check/model_backwards_compat_inference.py
##
@@ -0,0 +1,168 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from common import *
+
+def test_module_checkpoint_api():
+ model_name = 'module_checkpoint_api'
+ print ('Performing inference for model/API %s' %model_name)
+ data = download_data_from_s3(model_name)
+ test_data = data['data']
+ test_label = data['labels']
+
+ data_iter = mx.io.NDArrayIter(test_data, test_label, batch_size=10)
+
+ ## For each MXNet version that has the saved models
+ for folder in get_top_level_folders_in_bucket(s3, model_bucket_name):
+ bucket = s3.Bucket(model_bucket_name)
+ prefix = folder + backslash + model_name
+ model_files_meta = list(bucket.objects.filter(Prefix = prefix))
+ if len(model_files_meta) == 0:
+ print ('No trained models found under path : %s' %prefix)
+ continue
+ model_files = list()
+ ## For each file found under a model folder :
+ for obj in model_files_meta:
+ file_name = obj.key.split('/')[2]
+ model_files.append(file_name)
+ ## Download this file---
+ bucket.download_file(obj.key, file_name)
Review comment:
where this it download the file to?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r202281142 ## File path: tests/nightly/model_backwards_compatibility_check/common.py ## @@ -0,0 +1,199 @@ +#!/usr/bin/env python + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + + +import boto3 +import mxnet as mx +import json +import os +import numpy as np +import logging +from mxnet import nd, autograd, gluon +import mxnet.ndarray as nd +from mxnet.gluon.data.vision import transforms, datasets +from mxnet import autograd as ag +import mxnet.ndarray as F +from mxnet.gluon import nn, rnn +import re +import time +import sys +from mxnet.test_utils import assert_almost_equal + +# Set fixed random seeds. +mx.random.seed(7, ctx = mx.cpu()) +np.random.seed(7) +logging.getLogger().setLevel(logging.DEBUG) + +# get the current mxnet version we are running on +mxnet_version = mx.__version__ +model_bucket_name = 'mxnet-model-backwards-compatibility-models' +data_bucket_name = 'mxnet-model-backwards-compatibility-data' Review comment: Do we really need two separate buckets? I'd prefer if we could only use one to reduce the maintenance overhead. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r202280862
##
File path: ci/docker/runtime_functions.sh
##
@@ -854,6 +854,27 @@ nightly_test_javascript() {
make -C /work/mxnet/amalgamation libmxnet_predict.js MIN=1
EMCC=/work/deps/emscripten/emcc
}
+#Tests Model backwards compatibility on MXNet
+nightly_model_backwards_compat_test() {
+set -ex
+export PYTHONPATH=./python/
+pip install boto3
+
./tests/nightly/model_backwards_compatibility_check/model_backward_compat_checker.sh
+}
+
+#Backfills S3 bucket with models trained on earlier versions of mxnet
+nightly_model_backwards_compat_train() {
+set -ex
+export PYTHONPATH=./python/
+VENV=mbcc_py2_venv
+virtualenv -p `which python2` $VENV
+source $VENV/bin/activate
+pip install boto3
Review comment:
Please note (and add a comment or move it somewhere else) that this script
can not be called within our regular docker environment because the IAM roles
do not get propagated into the container as of now. Also, we have security
restrictions here that will have to be addressed before we can bring this into
production. I'll cover that later.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
marcoabreu commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r202280556 ## File path: tests/nightly/model_backwards_compatibility_check/JenkinsfileForMBCC ## @@ -0,0 +1,79 @@ +// -*- mode: groovy -*- +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + + +//This is a Jenkinsfile for the model backwards compatibility checker. The format and some functions have been picked up from the top-level Jenkinsfile. Review comment: For now, we will have this in a separate Jenkinsfile for testing purposes. In future, I'd like to make this part of the regular Jenkins nightly pipeline. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
