[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r206347507
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,82 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+set -ex
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version --user
+}
+
+## Cuts the string and gives only the major version part.
+## eg : 12.3.0 ---> 12
+get_major_version() {
+major=$(echo $1 | cut -d. -f1)
+echo $major
+}
+
+## We read the current major version from libinfo.py file. And we extract the
major version from it.
+curr_mxnet_version=$(grep -w "__version__" python/mxnet/libinfo.py | grep -o
'".*"' | sed 's/"//g')
+curr_major_version=$(get_major_version $curr_mxnet_version)
Review comment:
Done. ca01aa2bd9a183af10a4c3ae7ca5194c395b4ffb.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r206344952
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version
+}
+
+install_boto3(){
+ echo "Intalling boto3"
+ pip install boto3
+}
+
+echo `pwd`
+cd tests/nightly/model_backwards_compatibility_check
+echo `pwd`
+
+install_boto3
+
+## Fetch the latest release tags, filtering out 'rcs' and filtering out some
other irrelevant ones
+## This list is sorted in descending order chronologically. Keeping n = 5 for
a precautionary check.
+## Sample output for the below git tag command is : 1.2.0 utils 1.1.0 1.0.0
0.12.1
+previous_versions=($(git tag --sort=-creatordate | grep --invert-match rc |
head -n 5))
+count=0
+for version in ${previous_versions[*]}
+do
+ # We just need to train the previous two versions. This logic can be
changed later on as welll.
+ if [[ "$count" -gt 1 ]]
+ then
+ echo "Successfully trained files for the previous two MXNet
release versions"
+ exit 1
+ fi
+
+ ## If MXNet major version starts with a number >=1. with a wildcard
match for the minor version numbers
Review comment:
It handles here : d8fa75d72bd87466112f58865cebb38883b63171.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r206344664
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+set -ex
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version --user
+}
+
+echo `pwd`
+cd tests/nightly/model_backwards_compatibility_check
+echo `pwd`
+
+## Fetch the latest release tags, filtering out 'rcs' and filtering out some
other irrelevant ones
+## This list is sorted in descending order chronologically.
+## Sample output for the below git tag command is : 1.2.0 utils 1.1.0 1.0.0
0.12.1
+## so from this sample we will pick up the top two : 1.2.0 and 1.1.0 and train
models on them
+## Now while performing inference the latest version could be 1.3.0, which
will help in validating models trained
+## on 1.1.0 and 1.2.0 by loading them on the latest version (1.3.0)
+## Over a period of time, the model repository will grow since with every new
release we
+## upload models trained on newer versions as well through this script
+previous_versions=($(git tag --sort=-creatordate | grep --invert-match rc))
+count=0
+for version in ${previous_versions[*]}
+do
+ # We just need to train the previous two versions. This logic can be
changed later on as welll.
+ if [[ "$count" -gt 1 ]]
+ then
+ echo "Successfully trained files for the previous two MXNet
release versions"
+ exit 0
+ fi
+
+ ## If MXNet major version starts with a number >=1. with a wildcard
match for the minor version numbers
+ ## Could have used a [[:digit:]]+. as well but it was not working as a
traditional regex in bash.
+ ## so had to resort to using [[:digit:]] [[:digit:]]* to indicate
multi-digit version regex match
+ if [[ $version = [[:digit:][[:digit:]]*.[[:digit:]].[[:digit:]] ]]
Review comment:
Done. d8fa75d72bd87466112f58865cebb38883b63171.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r206344638
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version
+}
+
+install_boto3(){
+ echo "Intalling boto3"
+ pip install boto3
+}
+
+echo `pwd`
+cd tests/nightly/model_backwards_compatibility_check
+echo `pwd`
+
+install_boto3
+
+## Fetch the latest release tags, filtering out 'rcs' and filtering out some
other irrelevant ones
+## This list is sorted in descending order chronologically. Keeping n = 5 for
a precautionary check.
+## Sample output for the below git tag command is : 1.2.0 utils 1.1.0 1.0.0
0.12.1
+previous_versions=($(git tag --sort=-creatordate | grep --invert-match rc |
head -n 5))
+count=0
+for version in ${previous_versions[*]}
+do
+ # We just need to train the previous two versions. This logic can be
changed later on as welll.
+ if [[ "$count" -gt 1 ]]
Review comment:
Done. d8fa75d72bd87466112f58865cebb38883b63171.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r206270486 ## File path: tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh ## @@ -57,10 +57,10 @@ do fi ## If MXNet major version starts with a number >=1. with a wildcard match for the minor version numbers - if [[ $version = [1-9]* ]] + if [[ $version = [1-9].[0-9].[0-9] ]] Review comment: Good point. Done 16d320a926d4f23265f730d2fc197ecec5bc4a2e. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r206270517 ## File path: tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh ## @@ -0,0 +1,67 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +#Author: Piyush Ghai + Review comment: Done ceac705fe133cc7100f8c0b5076f943c4315462d This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r206270500 ## File path: tests/nightly/model_backwards_compatibility_check/model_backward_compat_checker.sh ## @@ -0,0 +1,28 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +#Author: Piyush Ghai + Review comment: Done ceac705fe133cc7100f8c0b5076f943c4315462d. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r206270517 ## File path: tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh ## @@ -0,0 +1,67 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +#Author: Piyush Ghai + Review comment: Done ceac705fe133cc7100f8c0b5076f943c4315462d. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r206270500 ## File path: tests/nightly/model_backwards_compatibility_check/model_backward_compat_checker.sh ## @@ -0,0 +1,28 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +#Author: Piyush Ghai + Review comment: Done ceac705fe133cc7100f8c0b5076f943c4315462d This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r206270486 ## File path: tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh ## @@ -57,10 +57,10 @@ do fi ## If MXNet major version starts with a number >=1. with a wildcard match for the minor version numbers - if [[ $version = [1-9]* ]] + if [[ $version = [1-9].[0-9].[0-9] ]] Review comment: Good point. Done 16d320a926d4f23265f730d2fc197ecec5bc4a2e This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r206205042 ## File path: tests/nightly/model_backwards_compatibility_check/common.py ## @@ -42,7 +42,7 @@ # get the current mxnet version we are running on mxnet_version = mx.__version__ -model_bucket_name = 'mxnet-model-backwards-compatibility-models' +model_bucket_name = 'mxnet-ci-prod-backwards-compatibility-models' Review comment: This bucket name change also has to be made in upload_models_to_s3.sh. Done that in 33096c02f56186464b34ba6b88480290eb89263f This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205518069
##
File path: ci/docker/runtime_functions.sh
##
@@ -895,6 +895,25 @@ nightly_test_javascript() {
make -C /work/mxnet/amalgamation libmxnet_predict.js MIN=1
EMCC=/work/deps/emscripten/emcc
}
+#Tests Model backwards compatibility on MXNet
+nightly_model_backwards_compat_test() {
+set -ex
+export PYTHONPATH=./python/
+
./tests/nightly/model_backwards_compatibility_check/model_backward_compat_checker.sh
+}
+
+#Backfills S3 bucket with models trained on earlier versions of mxnet
+nightly_model_backwards_compat_train() {
+set -ex
+export PYTHONPATH=./python/
+VENV=mbcc_py2_venv
+virtualenv -p `which python2` $VENV
Review comment:
Ah I see. Fixed in edd68164526cf591b090322f8f6e0805e17c17ee
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205518052
##
File path: tests/nightly/model_backwards_compatibility_check/JenkinsfileForMBCC
##
@@ -0,0 +1,86 @@
+// -*- mode: groovy -*-
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+//This is a Jenkinsfile for the model backwards compatibility checker. The
format and some functions have been picked up from the top-level Jenkinsfile.
+
+err = null
+
+def init_git() {
+ deleteDir()
+ retry(5) {
+try {
+ timeout(time: 15, unit: 'MINUTES') {
+checkout scm
+sh 'git submodule update --init --recursive'
+sh 'git clean -d -f'
+ }
+} catch (exc) {
+ deleteDir()
+ error "Failed to fetch source codes with ${exc}"
+ sleep 2
+}
+ }
+}
+
+def docker_run(platform, function_name, use_nvidia, shared_mem = '500m') {
+ def command = "ci/build.py --docker-registry ${env.DOCKER_CACHE_REGISTRY}
%USE_NVIDIA% --platform %PLATFORM% --shm-size %SHARED_MEM%
/work/runtime_functions.sh %FUNCTION_NAME%"
+ command = command.replaceAll('%USE_NVIDIA%', use_nvidia ? '--nvidiadocker' :
'')
+ command = command.replaceAll('%PLATFORM%', platform)
+ command = command.replaceAll('%FUNCTION_NAME%', function_name)
+ command = command.replaceAll('%SHARED_MEM%', shared_mem)
+
+ sh command
+}
+
+try {
+ stage('MBCC'){
+parallel 'ModelBackwardsCompat: CPU': {
+ node('mxnetlinux-cpu') {
+ws('workspace/modelBackwardsCompat') {
+
+init_git()
+// Train models on older versions
+docker_run('ubuntu_nightly_cpu',
'nightly_model_backwards_compat_train', false)
+// upload files to S3 here outside of the docker environment
+sh
/tests/nightly/model_backwards_compatibility_check/upload_models_to_s3.sh
+// Perform inference on these models
+docker_run('ubuntu_nightly_cpu',
'nightly_model_backwards_compat_test', false)
+
+ }
Review comment:
Done. 87103d43dee19bb42ac4a7296a9be940ca44dffc
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205518026
##
File path: tests/nightly/model_backwards_compatibility_check/JenkinsfileForMBCC
##
@@ -0,0 +1,86 @@
+// -*- mode: groovy -*-
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+//This is a Jenkinsfile for the model backwards compatibility checker. The
format and some functions have been picked up from the top-level Jenkinsfile.
+
+err = null
+
+def init_git() {
+ deleteDir()
+ retry(5) {
+try {
+ timeout(time: 15, unit: 'MINUTES') {
+checkout scm
+sh 'git submodule update --init --recursive'
+sh 'git clean -d -f'
+ }
+} catch (exc) {
+ deleteDir()
+ error "Failed to fetch source codes with ${exc}"
+ sleep 2
+}
+ }
+}
+
+def docker_run(platform, function_name, use_nvidia, shared_mem = '500m') {
+ def command = "ci/build.py --docker-registry ${env.DOCKER_CACHE_REGISTRY}
%USE_NVIDIA% --platform %PLATFORM% --shm-size %SHARED_MEM%
/work/runtime_functions.sh %FUNCTION_NAME%"
+ command = command.replaceAll('%USE_NVIDIA%', use_nvidia ? '--nvidiadocker' :
'')
+ command = command.replaceAll('%PLATFORM%', platform)
+ command = command.replaceAll('%FUNCTION_NAME%', function_name)
+ command = command.replaceAll('%SHARED_MEM%', shared_mem)
+
+ sh command
+}
+
+try {
+ stage('MBCC'){
+parallel 'ModelBackwardsCompat: CPU': {
+ node('mxnetlinux-cpu') {
+ws('workspace/modelBackwardsCompat') {
+
+init_git()
+// Train models on older versions
+docker_run('ubuntu_nightly_cpu',
'nightly_model_backwards_compat_train', false)
Review comment:
Done. 87103d43dee19bb42ac4a7296a9be940ca44dffc
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205518042
##
File path: tests/nightly/model_backwards_compatibility_check/JenkinsfileForMBCC
##
@@ -0,0 +1,86 @@
+// -*- mode: groovy -*-
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+//This is a Jenkinsfile for the model backwards compatibility checker. The
format and some functions have been picked up from the top-level Jenkinsfile.
+
+err = null
+
+def init_git() {
+ deleteDir()
+ retry(5) {
+try {
+ timeout(time: 15, unit: 'MINUTES') {
+checkout scm
+sh 'git submodule update --init --recursive'
+sh 'git clean -d -f'
+ }
+} catch (exc) {
+ deleteDir()
+ error "Failed to fetch source codes with ${exc}"
+ sleep 2
+}
+ }
+}
+
+def docker_run(platform, function_name, use_nvidia, shared_mem = '500m') {
+ def command = "ci/build.py --docker-registry ${env.DOCKER_CACHE_REGISTRY}
%USE_NVIDIA% --platform %PLATFORM% --shm-size %SHARED_MEM%
/work/runtime_functions.sh %FUNCTION_NAME%"
+ command = command.replaceAll('%USE_NVIDIA%', use_nvidia ? '--nvidiadocker' :
'')
+ command = command.replaceAll('%PLATFORM%', platform)
+ command = command.replaceAll('%FUNCTION_NAME%', function_name)
+ command = command.replaceAll('%SHARED_MEM%', shared_mem)
+
+ sh command
+}
+
+try {
+ stage('MBCC'){
+parallel 'ModelBackwardsCompat: CPU': {
+ node('mxnetlinux-cpu') {
+ws('workspace/modelBackwardsCompat') {
+
+init_git()
+// Train models on older versions
+docker_run('ubuntu_nightly_cpu',
'nightly_model_backwards_compat_train', false)
+// upload files to S3 here outside of the docker environment
+sh
/tests/nightly/model_backwards_compatibility_check/upload_models_to_s3.sh
Review comment:
Good catch. Added a '.' before /tests.
Done in 87103d43dee19bb42ac4a7296a9be940ca44dffc.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205299373
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version
+}
+
+install_boto3(){
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205299356
##
File path: ci/docker/runtime_functions.sh
##
@@ -854,6 +854,29 @@ nightly_test_javascript() {
make -C /work/mxnet/amalgamation libmxnet_predict.js MIN=1
EMCC=/work/deps/emscripten/emcc
}
+#Tests Model backwards compatibility on MXNet
+nightly_model_backwards_compat_test() {
+set -ex
+export PYTHONPATH=./python/
+pip install boto3
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205299184
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version
+}
+
+install_boto3(){
+ echo "Intalling boto3"
+ pip install boto3
+}
+
+echo `pwd`
+cd tests/nightly/model_backwards_compatibility_check
+echo `pwd`
+
+install_boto3
+
+## Fetch the latest release tags, filtering out 'rcs' and filtering out some
other irrelevant ones
+## This list is sorted in descending order chronologically. Keeping n = 5 for
a precautionary check.
+## Sample output for the below git tag command is : 1.2.0 utils 1.1.0 1.0.0
0.12.1
+previous_versions=($(git tag --sort=-creatordate | grep --invert-match rc |
head -n 5))
+count=0
+for version in ${previous_versions[*]}
+do
+ # We just need to train the previous two versions. This logic can be
changed later on as welll.
+ if [[ "$count" -gt 1 ]]
Review comment:
I have updated the readme and added more comments to this file in :
1f63941e9ade0eaa2f4361ed5877bc97defcf6fb
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205299025
##
File path: ci/docker/runtime_functions.sh
##
@@ -854,6 +854,27 @@ nightly_test_javascript() {
make -C /work/mxnet/amalgamation libmxnet_predict.js MIN=1
EMCC=/work/deps/emscripten/emcc
}
+#Tests Model backwards compatibility on MXNet
+nightly_model_backwards_compat_test() {
+set -ex
+export PYTHONPATH=./python/
+pip install boto3
+
./tests/nightly/model_backwards_compatibility_check/model_backward_compat_checker.sh
+}
+
+#Backfills S3 bucket with models trained on earlier versions of mxnet
+nightly_model_backwards_compat_train() {
+set -ex
+export PYTHONPATH=./python/
+VENV=mbcc_py2_venv
+virtualenv -p `which python2` $VENV
+source $VENV/bin/activate
+pip install boto3
Review comment:
Done in f7a36eb7bdd4836c6e0ed6ee58f199d35320f06b
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205299025
##
File path: ci/docker/runtime_functions.sh
##
@@ -854,6 +854,27 @@ nightly_test_javascript() {
make -C /work/mxnet/amalgamation libmxnet_predict.js MIN=1
EMCC=/work/deps/emscripten/emcc
}
+#Tests Model backwards compatibility on MXNet
+nightly_model_backwards_compat_test() {
+set -ex
+export PYTHONPATH=./python/
+pip install boto3
+
./tests/nightly/model_backwards_compatibility_check/model_backward_compat_checker.sh
+}
+
+#Backfills S3 bucket with models trained on earlier versions of mxnet
+nightly_model_backwards_compat_train() {
+set -ex
+export PYTHONPATH=./python/
+VENV=mbcc_py2_venv
+virtualenv -p `which python2` $VENV
+source $VENV/bin/activate
+pip install boto3
Review comment:
Done in 1f63941e9ade0eaa2f4361ed5877bc97defcf6fb
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205201337
##
File path: ci/docker/runtime_functions.sh
##
@@ -854,6 +854,27 @@ nightly_test_javascript() {
make -C /work/mxnet/amalgamation libmxnet_predict.js MIN=1
EMCC=/work/deps/emscripten/emcc
}
+#Tests Model backwards compatibility on MXNet
+nightly_model_backwards_compat_test() {
+set -ex
+export PYTHONPATH=./python/
+pip install boto3
+
./tests/nightly/model_backwards_compatibility_check/model_backward_compat_checker.sh
+}
+
+#Backfills S3 bucket with models trained on earlier versions of mxnet
+nightly_model_backwards_compat_train() {
+set -ex
+export PYTHONPATH=./python/
+VENV=mbcc_py2_venv
+virtualenv -p `which python2` $VENV
+source $VENV/bin/activate
+pip install boto3
Review comment:
Well, now that I have another look at
https://github.com/apache/incubator-mxnet/blob/master/tests/nightly/broken_link_checker_test/JenkinsfileForBLC#L57
I understand the interaction with S3.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205201337
##
File path: ci/docker/runtime_functions.sh
##
@@ -854,6 +854,27 @@ nightly_test_javascript() {
make -C /work/mxnet/amalgamation libmxnet_predict.js MIN=1
EMCC=/work/deps/emscripten/emcc
}
+#Tests Model backwards compatibility on MXNet
+nightly_model_backwards_compat_test() {
+set -ex
+export PYTHONPATH=./python/
+pip install boto3
+
./tests/nightly/model_backwards_compatibility_check/model_backward_compat_checker.sh
+}
+
+#Backfills S3 bucket with models trained on earlier versions of mxnet
+nightly_model_backwards_compat_train() {
+set -ex
+export PYTHONPATH=./python/
+VENV=mbcc_py2_venv
+virtualenv -p `which python2` $VENV
+source $VENV/bin/activate
+pip install boto3
Review comment:
Well, now that I have another look at
https://github.com/apache/incubator-mxnet/blob/master/tests/nightly/broken_link_checker_test/JenkinsfileForBLC#L57
I get the interaction with S3.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205199709
##
File path: ci/docker/runtime_functions.sh
##
@@ -854,6 +854,27 @@ nightly_test_javascript() {
make -C /work/mxnet/amalgamation libmxnet_predict.js MIN=1
EMCC=/work/deps/emscripten/emcc
}
+#Tests Model backwards compatibility on MXNet
+nightly_model_backwards_compat_test() {
+set -ex
+export PYTHONPATH=./python/
+pip install boto3
+
./tests/nightly/model_backwards_compatibility_check/model_backward_compat_checker.sh
+}
+
+#Backfills S3 bucket with models trained on earlier versions of mxnet
+nightly_model_backwards_compat_train() {
+set -ex
+export PYTHONPATH=./python/
+VENV=mbcc_py2_venv
+virtualenv -p `which python2` $VENV
+source $VENV/bin/activate
+pip install boto3
Review comment:
I'm a little confused about it. So this is indeed a couple of lines of shell
script which can be easily added in the docker file for MBCC, but how do I
install boto on the Jenkins machines since the training script would not be
running inside a docker container ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205198798
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version
+}
+
+install_boto3(){
+ echo "Intalling boto3"
+ pip install boto3
+}
+
+echo `pwd`
+cd tests/nightly/model_backwards_compatibility_check
+echo `pwd`
+
+install_boto3
+
+## Fetch the latest release tags, filtering out 'rcs' and filtering out some
other irrelevant ones
+## This list is sorted in descending order chronologically. Keeping n = 5 for
a precautionary check.
+## Sample output for the below git tag command is : 1.2.0 utils 1.1.0 1.0.0
0.12.1
+previous_versions=($(git tag --sort=-creatordate | grep --invert-match rc |
head -n 5))
+count=0
+for version in ${previous_versions[*]}
+do
+ # We just need to train the previous two versions. This logic can be
changed later on as welll.
+ if [[ "$count" -gt 1 ]]
Review comment:
This line -- "previous_versions=($(git tag --sort=-creatordate | grep
--invert-match rc | head -n 5))" is version agnostic, which means even if you
change the major version, the models will still be trained.
With multiple minor versions, the training will happen on all of the tagged
git versions.
So essentially, whenever a new release (be it major version/minor version)
comes out, it will be pulled in using the git-tag command and the training file
will save a bunch of models for it in S3.
Please let me know if there's still some confusion here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205188108
##
File path: ci/docker/runtime_functions.sh
##
@@ -854,6 +854,29 @@ nightly_test_javascript() {
make -C /work/mxnet/amalgamation libmxnet_predict.js MIN=1
EMCC=/work/deps/emscripten/emcc
}
+#Tests Model backwards compatibility on MXNet
+nightly_model_backwards_compat_test() {
+set -ex
+export PYTHONPATH=./python/
+pip install boto3
Review comment:
Sicne boto is easily available as a pip package, I think it will be better
placed here :
https://github.com/apache/incubator-mxnet/blob/master/ci/docker/install/ubuntu_python.sh
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r205173761
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version
+}
+
+install_boto3(){
+ echo "Intalling boto3"
+ pip install boto3
+}
+
+echo `pwd`
+cd tests/nightly/model_backwards_compatibility_check
+echo `pwd`
+
+install_boto3
+
+## Fetch the latest release tags, filtering out 'rcs' and filtering out some
other irrelevant ones
+## This list is sorted in descending order chronologically. Keeping n = 5 for
a precautionary check.
+## Sample output for the below git tag command is : 1.2.0 utils 1.1.0 1.0.0
0.12.1
+previous_versions=($(git tag --sort=-creatordate | grep --invert-match rc |
head -n 5))
+count=0
+for version in ${previous_versions[*]}
+do
+ # We just need to train the previous two versions. This logic can be
changed later on as welll.
+ if [[ "$count" -gt 1 ]]
Review comment:
Good point and it's been taken care of in the inference scripts. They pull
off all the model files (they don't filter out on which version files were
trained on). So in a way, after a couple of releases the training script would
have built a pool of all the previous releases for the inference files to work
with.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r203456051
##
File path: ci/docker/runtime_functions.sh
##
@@ -854,6 +854,27 @@ nightly_test_javascript() {
make -C /work/mxnet/amalgamation libmxnet_predict.js MIN=1
EMCC=/work/deps/emscripten/emcc
}
+#Tests Model backwards compatibility on MXNet
+nightly_model_backwards_compat_test() {
+set -ex
+export PYTHONPATH=./python/
+pip install boto3
+
./tests/nightly/model_backwards_compatibility_check/model_backward_compat_checker.sh
+}
+
+#Backfills S3 bucket with models trained on earlier versions of mxnet
+nightly_model_backwards_compat_train() {
+set -ex
+export PYTHONPATH=./python/
+VENV=mbcc_py2_venv
+virtualenv -p `which python2` $VENV
+source $VENV/bin/activate
+pip install boto3
Review comment:
I added a comment indicating that it should not be called from the regular
docker environment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r203455581
##
File path: tests/nightly/model_backwards_compatibility_check/common.py
##
@@ -0,0 +1,210 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import boto3
+import mxnet as mx
+import json
+import os
+import numpy as np
+import logging
+from mxnet import nd, autograd, gluon
+import mxnet.ndarray as nd
+from mxnet.gluon.data.vision import transforms, datasets
+from mxnet import autograd as ag
+import mxnet.ndarray as F
+from mxnet.gluon import nn, rnn
+import re
+import time
+import sys
+from mxnet.test_utils import assert_almost_equal
+
+# Set fixed random seeds.
+mx.random.seed(7)
+np.random.seed(7)
+logging.getLogger().setLevel(logging.DEBUG)
+
+# get the current mxnet version we are running on
+mxnet_version = mx.__version__
+model_bucket_name = 'mxnet-model-backwards-compatibility-models'
+data_folder = 'mxnet-model-backwards-compatibility-data'
+backslash = '/'
+s3 = boto3.resource('s3')
+ctx = mx.cpu(0)
+
+def get_module_api_model_definition():
+input = mx.symbol.Variable('data')
+input = mx.symbol.Flatten(data=input)
+
+fc1 = mx.symbol.FullyConnected(data=input, name='fc1', num_hidden=128)
+act1 = mx.sym.Activation(data=fc1, name='relu1', act_type="relu")
+fc2 = mx.symbol.FullyConnected(data=fc1, name='fc2', num_hidden=2)
+op = mx.symbol.SoftmaxOutput(data=fc2, name='softmax')
+model = mx.mod.Module(symbol=op, context=ctx, data_names=['data'],
label_names=['softmax_label'])
+return model
+
+def save_inference_results(inference_results, model_name):
+assert (isinstance(inference_results, mx.ndarray.ndarray.NDArray))
+mx.nd.save(model_name + '-inference', {'inference' : inference_results})
+
+def load_inference_results(model_name):
+inf_dict = mx.nd.load(model_name+'-inference')
+return inf_dict['inference']
+
+def save_data_and_labels(test_data, test_labels, model_name):
+assert (isinstance(test_data, mx.ndarray.ndarray.NDArray))
+assert (isinstance(test_labels, mx.ndarray.ndarray.NDArray))
+mx.nd.save(model_name + '-data', {'data' : test_data, 'labels' :
test_labels})
+
+def upload_data_and_labels_to_s3(model_name):
+s3 = boto3.client('s3')
+file = model_name + '-data'
+s3.upload_file(file, model_bucket_name, data_folder + backslash + file)
+print ('data files successfully uploaded to s3')
+
+def upload_model_files_to_s3(files, folder_name):
+s3 = boto3.client('s3')
+for file in files:
+s3.upload_file(file, model_bucket_name, folder_name + file)
+
+def clean_model_files(files, model_name):
+files.append(model_name + '-inference')
+files.append(model_name + '-data')
+
+for file in files:
+if os.path.isfile(file):
+os.remove(file)
+
+def download_data_from_s3(model_name):
+print ('Downloading data files for %s from bucket %s'%(model_name,
model_bucket_name + backslash + data_folder))
+bucket = s3.Bucket(model_bucket_name)
+prefix = data_folder + backslash + model_name + '-data'
+data_files_meta = list(bucket.objects.filter(Prefix = prefix))
+if len(data_files_meta) == 0:
+print ('No data files found for %s' %model_name)
Review comment:
Done. 9d4d0993b56ec8b3148e1a0ee90c81424bb1f0cb
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r203447235 ## File path: tests/nightly/model_backwards_compatibility_check/model_backward_compat_checker.sh ## @@ -0,0 +1,28 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +#Author: Piyush Ghai + +echo "Invoking model_backwards_compat_checker.sh script" +echo `pwd` +cd tests/nightly/model_backwards_compatibility_check +echo `pwd` + +echo '==' +python model_backwards_compat_inference.py Review comment: runtime_functions calls the bash script -- 'model_backward_compat_checker.sh' This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r203446344
##
File path:
tests/nightly/model_backwards_compatibility_check/model_backwards_compat_inference.py
##
@@ -0,0 +1,149 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from common import *
+
+def test_module_checkpoint_api():
+ model_name = 'module_checkpoint_api'
+ print ('Performing inference for model/API %s' %model_name)
+ data = download_data_from_s3(model_name)
+ if data is None:
+ print ('No data files found for %s' %model_name)
+ return
+
+ test_data = data['data']
+ test_label = data['labels']
+
+ data_iter = mx.io.NDArrayIter(test_data, test_label, batch_size=10)
+
+ ## For each MXNet version that has the saved models
+ for folder in get_top_level_folders_in_bucket(s3, model_bucket_name):
+ print ('Fetching files for MXNet version : %s and model %s'
%(folder, model_name))
+ model_files = download_model_files_from_s3(model_name, folder)
+ if len(model_files) == 0:
+ print('No training files found for %s for MXNet version
: %s'%(model_name, folder))
+ continue
+ ## Load the model and perform inference
+ loaded_model = get_module_api_model_definition()
+
+ sym, arg_params, aux_params =
mx.model.load_checkpoint(model_name, 1)
+ loaded_model.bind(data_shapes=data_iter.provide_data,
label_shapes=data_iter.provide_label)
+ loaded_model.set_params(arg_params, aux_params)
+
+ old_inference_results = load_inference_results(model_name)
+ inference_results = loaded_model.predict(data_iter)
+ ## Check whether they are equal or not ?
+ assert_almost_equal(inference_results.asnumpy(),
old_inference_results.asnumpy())
+ clean_model_files(model_files, model_name)
+ print ('=')
+
+ print ('Assertion passed for model : %s' %model_name)
+
+
+def test_lenet_gluon_load_params_api():
+ model_name = 'lenet_gluon_save_params_api'
+ print ('Performing inference for model/API %s' %model_name)
+ ## Get data from S3
+ data = download_data_from_s3(model_name)
+ if data is None:
+ print ('No data files found for %s' %model_name)
+ return
+
+ test_data = data['data']
+
+ for folder in get_top_level_folders_in_bucket(s3, model_bucket_name):
+ print ('Fetching files for MXNet version : %s and model %s'
%(folder, model_name))
+ model_files = download_model_files_from_s3(model_name, folder)
+ if len(model_files) == 0:
+ print('No training files found for %s for MXNet version
: %s'%(model_name, folder))
+ continue
+ ## Load the model and perform inference
+ loaded_model = Net()
+ loaded_model.load_params(model_name+'-params')
+ output = loaded_model(test_data)
+ old_inference_results = mx.nd.load(model_name +
'-inference')['inference']
+ assert_almost_equal(old_inference_results.asnumpy(),
output.asnumpy())
+ clean_model_files(model_files, model_name)
+ print ('=')
+ print ('Assertion passed for model : %s' %model_name)
+
+def test_lenet_gluon_hybrid_imports_api():
Review comment:
We would still need to instantiate the models and load them using different
loading APIs. Hence the boiler plate code cannot be done away with. I have
tried to keep the boiler plate as small as possible to perform the inference.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r203445850
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version
+}
+
+install_boto3(){
+ echo "Intalling boto3"
+ pip install boto3
+}
+
+echo `pwd`
+cd tests/nightly/model_backwards_compatibility_check
+echo `pwd`
+
+install_boto3
+
+## Fetch the latest release tags, filtering out 'rcs' and filtering out some
other irrelevant ones
+## This list is sorted in descending order chronologically. Keeping n = 5 for
a precautionary check.
+## Sample output for the below git tag command is : 1.2.0 utils 1.1.0 1.0.0
0.12.1
+previous_versions=($(git tag --sort=-creatordate | grep --invert-match rc |
head -n 5))
+count=0
+for version in ${previous_versions[*]}
+do
+ # We just need to train the previous two versions. This logic can be
changed later on as welll.
+ if [[ "$count" -gt 1 ]]
+ then
+ echo "Successfully trained files for the previous two MXNet
release versions"
+ exit 1
+ fi
+
+ ## If MXNet major version starts with a number >=1. with a wildcard
match for the minor version numbers
Review comment:
The regex below filters out the versions. Also, since the git tag command is
giving results in descending order chronologically, we get the latest set of
tags. When I was running the git tag command, I found couple of tags with names
such as "utils" which do not seem to be a release version IMO. So I added this
regex to filter out release versions starting with a number.
I already have added comments for this before the for loop above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r203444875
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version
+}
+
+install_boto3(){
+ echo "Intalling boto3"
+ pip install boto3
+}
+
+echo `pwd`
+cd tests/nightly/model_backwards_compatibility_check
+echo `pwd`
+
+install_boto3
+
+## Fetch the latest release tags, filtering out 'rcs' and filtering out some
other irrelevant ones
+## This list is sorted in descending order chronologically. Keeping n = 5 for
a precautionary check.
+## Sample output for the below git tag command is : 1.2.0 utils 1.1.0 1.0.0
0.12.1
+previous_versions=($(git tag --sort=-creatordate | grep --invert-match rc |
head -n 5))
+count=0
+for version in ${previous_versions[*]}
+do
+ # We just need to train the previous two versions. This logic can be
changed later on as welll.
+ if [[ "$count" -gt 1 ]]
Review comment:
We are performing training of all the model files on the previous two
released versions of MXNet.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r202846947
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version
+}
+
+install_boto3(){
+ echo "Intalling boto3"
+ pip install boto3
+}
+
+echo `pwd`
+cd tests/nightly/model_backwards_compatibility_check
+echo `pwd`
+
+install_boto3
+
+install_mxnet 1.1.0
Review comment:
Done. e764d5a2898d42f263688685bf575234a492a650
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r202846856 ## File path: tests/nightly/model_backwards_compatibility_check/common.py ## @@ -0,0 +1,199 @@ +#!/usr/bin/env python + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + + +import boto3 +import mxnet as mx +import json +import os +import numpy as np +import logging +from mxnet import nd, autograd, gluon +import mxnet.ndarray as nd +from mxnet.gluon.data.vision import transforms, datasets +from mxnet import autograd as ag +import mxnet.ndarray as F +from mxnet.gluon import nn, rnn +import re +import time +import sys +from mxnet.test_utils import assert_almost_equal + +# Set fixed random seeds. +mx.random.seed(7, ctx = mx.cpu()) +np.random.seed(7) +logging.getLogger().setLevel(logging.DEBUG) + +# get the current mxnet version we are running on +mxnet_version = mx.__version__ +model_bucket_name = 'mxnet-model-backwards-compatibility-models' +data_bucket_name = 'mxnet-model-backwards-compatibility-data' Review comment: Done. c80ee31 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r202753141
##
File path: ci/docker/runtime_functions.sh
##
@@ -854,6 +854,27 @@ nightly_test_javascript() {
make -C /work/mxnet/amalgamation libmxnet_predict.js MIN=1
EMCC=/work/deps/emscripten/emcc
}
+#Tests Model backwards compatibility on MXNet
+nightly_model_backwards_compat_test() {
+set -ex
+export PYTHONPATH=./python/
+pip install boto3
+
./tests/nightly/model_backwards_compatibility_check/model_backward_compat_checker.sh
+}
+
+#Backfills S3 bucket with models trained on earlier versions of mxnet
+nightly_model_backwards_compat_train() {
+set -ex
+export PYTHONPATH=./python/
+VENV=mbcc_py2_venv
+virtualenv -p `which python2` $VENV
+source $VENV/bin/activate
+pip install boto3
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r202751868 ## File path: tests/nightly/model_backwards_compatibility_check/common.py ## @@ -0,0 +1,199 @@ +#!/usr/bin/env python + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + + +import boto3 +import mxnet as mx +import json +import os +import numpy as np +import logging +from mxnet import nd, autograd, gluon +import mxnet.ndarray as nd +from mxnet.gluon.data.vision import transforms, datasets +from mxnet import autograd as ag +import mxnet.ndarray as F +from mxnet.gluon import nn, rnn +import re +import time +import sys +from mxnet.test_utils import assert_almost_equal + +# Set fixed random seeds. +mx.random.seed(7, ctx = mx.cpu()) +np.random.seed(7) +logging.getLogger().setLevel(logging.DEBUG) + +# get the current mxnet version we are running on +mxnet_version = mx.__version__ +model_bucket_name = 'mxnet-model-backwards-compatibility-models' +data_bucket_name = 'mxnet-model-backwards-compatibility-data' Review comment: The bucket creation is a one-time job which I've already done so. Any other updates to the bucket will be done programatically only. I see your point though. I can however, create two folders under the same bucket. One folder for data and another for models. That should reduce the overhead. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r202751436
##
File path:
tests/nightly/model_backwards_compatibility_check/model_backwards_compat_inference.py
##
@@ -0,0 +1,168 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from common import *
+
+def test_module_checkpoint_api():
+ model_name = 'module_checkpoint_api'
+ print ('Performing inference for model/API %s' %model_name)
+ data = download_data_from_s3(model_name)
+ test_data = data['data']
+ test_label = data['labels']
+
+ data_iter = mx.io.NDArrayIter(test_data, test_label, batch_size=10)
+
+ ## For each MXNet version that has the saved models
+ for folder in get_top_level_folders_in_bucket(s3, model_bucket_name):
+ bucket = s3.Bucket(model_bucket_name)
+ prefix = folder + backslash + model_name
+ model_files_meta = list(bucket.objects.filter(Prefix = prefix))
+ if len(model_files_meta) == 0:
+ print ('No trained models found under path : %s' %prefix)
+ continue
+ model_files = list()
+ ## For each file found under a model folder :
+ for obj in model_files_meta:
+ file_name = obj.key.split('/')[2]
+ model_files.append(file_name)
+ ## Download this file---
+ bucket.download_file(obj.key, file_name)
Review comment:
This gets downloaded to the root folder of the mxnet-repo.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r202751125
##
File path:
tests/nightly/model_backwards_compatibility_check/model_backwards_compat_inference.py
##
@@ -0,0 +1,168 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from common import *
+
+def test_module_checkpoint_api():
+ model_name = 'module_checkpoint_api'
+ print ('Performing inference for model/API %s' %model_name)
+ data = download_data_from_s3(model_name)
+ test_data = data['data']
+ test_label = data['labels']
+
+ data_iter = mx.io.NDArrayIter(test_data, test_label, batch_size=10)
+
+ ## For each MXNet version that has the saved models
+ for folder in get_top_level_folders_in_bucket(s3, model_bucket_name):
+ bucket = s3.Bucket(model_bucket_name)
+ prefix = folder + backslash + model_name
+ model_files_meta = list(bucket.objects.filter(Prefix = prefix))
+ if len(model_files_meta) == 0:
+ print ('No trained models found under path : %s' %prefix)
+ continue
+ model_files = list()
+ ## For each file found under a model folder :
+ for obj in model_files_meta:
+ file_name = obj.key.split('/')[2]
+ model_files.append(file_name)
+ ## Download this file---
+ bucket.download_file(obj.key, file_name)
+
+ ## Load the model and perform inference
+ loaded_model = get_module_api_model_definition()
+
+ sym, arg_params, aux_params =
mx.model.load_checkpoint(model_name, 1)
+ loaded_model.bind(data_shapes=data_iter.provide_data,
label_shapes=data_iter.provide_label)
+ loaded_model.set_params(arg_params, aux_params)
+
+ old_inference_results = load_inference_results(model_name)
+ inference_results = loaded_model.predict(data_iter)
+ ## Check whether they are equal or not ?
+ assert_almost_equal(inference_results.asnumpy(),
old_inference_results.asnumpy())
+ clean_model_files(model_files, model_name)
+
+ print ('Assertion passed for model : %s' %model_name)
+
+
+def test_lenet_gluon_load_params_api():
+ model_name = 'lenet_gluon_save_params_api'
+ print ('Performing inference for model/API %s' %model_name)
+ ## Get data from S3
+ data = download_data_from_s3(model_name)
+
+ test_data = data['data']
+
+ for folder in get_top_level_folders_in_bucket(s3, model_bucket_name):
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r202751099
##
File path:
tests/nightly/model_backwards_compatibility_check/model_backwards_compat_train.py
##
@@ -0,0 +1,159 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from common import *
+
+def train_module_checkpoint_api():
+ model_name = 'module_checkpoint_api'
+ print ('Saving files for model %s' %model_name)
+ ### Prepare data
+ test_data = mx.nd.array(np.random.uniform(-1, 1, size=(20, 1)))
+ test_label = mx.nd.array(np.random.randint(0, 2, size=(20,)),
dtype='float32')
+ data_iter = mx.io.NDArrayIter(test_data, test_label, batch_size=10)
+
+
+ mod = get_module_api_model_definition()
+ mod.bind(data_shapes=data_iter.provide_data,
label_shapes=data_iter.provide_label)
+ weights = mx.initializer.Xavier(magnitude = 2.57)
+ mod.init_params(weights)
+
+ mod.save_checkpoint(model_name, 1)
+ ### Save the data, labels
+ save_data_and_labels(test_data, test_label, model_name)
+ upload_data_and_labels_to_s3(model_name)
+
+ inference_results = mod.predict(data_iter)
+ ### Save inference_results
+ save_inference_results(inference_results, model_name)
+ ### upload model and inference files to S3
+ files = list()
+ files.append(model_name + '-0001.params')
+ files.append(model_name + '-symbol.json')
+ files.append(model_name + '-inference')
+
+ mxnet_folder = str(mxnet_version) + backslash + model_name + backslash
+
+ upload_model_files_to_s3(files, mxnet_folder)
+
+ clean_model_files(files, model_name)
+
+def train_lenet_gluon_save_params_api():
+ model_name = 'lenet_gluon_save_params_api'
+ print ('Saving files for model %s' %model_name)
+ net = Net()
+ weights = mx.initializer.Xavier(magnitude = 2.57)
+ net.initialize(weights, ctx = [mx.cpu(0)])
+ ### Prepare data
+
+ test_data = mx.nd.array(np.random.uniform(-1, 1, size=(20, 1, 30, 30)))
+ output = net(test_data)
+ # print (y)
+ #### Save the test data as well.
+ #### Save the inference output ys
+ #### Save the model params
+
+ mx.nd.save(model_name + '-data', {'data' : test_data})
+ save_inference_results(output, model_name)
+ net.save_params(model_name + '-params')
+
+ mxnet_folder = str(mxnet_version) + backslash + model_name + backslash
+
+ files = list()
+ files.append(model_name + '-data')
+ files.append(model_name + '-inference')
+ files.append(model_name + '-params')
+
+ upload_data_and_labels_to_s3(model_name)
+
+ upload_model_files_to_s3(files, mxnet_folder)
+
+ clean_model_files(files, model_name)
+
+def train_lenet_gluon_hybrid_export_api():
+ model_name = 'lenet_gluon_hybrid_export_api'
+ print ('Saving files for model %s' %model_name)
+ net = HybridNet()
+ weights = mx.initializer.Xavier(magnitude = 2.57)
+ net.initialize(weights, ctx = [mx.cpu(0)])
+ net.hybridize()
+ ### Prepare data
+ test_data = mx.nd.array(np.random.uniform(-1, 1, size=(20, 1, 30, 30)))
+ output = net(test_data)
+ # print (y)
+### Save the test data as well.
+### Save the inference output ys
+### Save the model params
+
+ mx.nd.save(model_name + '-data', {'data' : test_data})
+ save_inference_results(output, model_name)
+ net.export(model_name, epoch=1)
+
+ mxnet_folder = str(mxnet_version) + backslash + model_name + backslash
+
+ files = list()
+ files.append(model_name + '-data')
+ files.append(model_name + '-inference')
+ files.append(model_name + '-0001.params')
+ files.append(model_name + '-symbol.json')
+
+
+ upload_data_and_labels_to_s3(model_name)
+
+ upload_model_files_to_s3(files, mxnet_folder)
+
+ clean_model_files(files, model_name)
+
+def train_lstm_gluon_save_parameters_api():
+ ## If this code is being run on version >= 1.2.0 only then execute it,
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r202749501
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python model_backwards_compat_train.py
+ echo '=='
+}
+
+install_mxnet() {
+ version=$1
+ echo "Installing MXNet "$version
+ pip install mxnet==$version
+}
+
+install_boto3(){
+ echo "Intalling boto3"
+ pip install boto3
+}
+
+echo `pwd`
+cd tests/nightly/model_backwards_compatibility_check
+echo `pwd`
+
+install_boto3
+
+install_mxnet 1.1.0
Review comment:
I'm not sure if I fully understood your comment here @marcoabreu . Git
branches generally have a lot of tags such 1.x.x.rc, 1.x.x.nightly etc... How
can we filter out only the actual releases from the git tags ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r202412563
##
File path:
tests/nightly/model_backwards_compatibility_check/model_backwards_compat_train.py
##
@@ -0,0 +1,159 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from common import *
+
+def train_module_checkpoint_api():
+ model_name = 'module_checkpoint_api'
+ print ('Saving files for model %s' %model_name)
+ ### Prepare data
+ test_data = mx.nd.array(np.random.uniform(-1, 1, size=(20, 1)))
+ test_label = mx.nd.array(np.random.randint(0, 2, size=(20,)),
dtype='float32')
+ data_iter = mx.io.NDArrayIter(test_data, test_label, batch_size=10)
+
+
+ mod = get_module_api_model_definition()
+ mod.bind(data_shapes=data_iter.provide_data,
label_shapes=data_iter.provide_label)
+ weights = mx.initializer.Xavier(magnitude = 2.57)
+ mod.init_params(weights)
+
+ mod.save_checkpoint(model_name, 1)
+ ### Save the data, labels
+ save_data_and_labels(test_data, test_label, model_name)
+ upload_data_and_labels_to_s3(model_name)
+
+ inference_results = mod.predict(data_iter)
+ ### Save inference_results
+ save_inference_results(inference_results, model_name)
+ ### upload model and inference files to S3
+ files = list()
+ files.append(model_name + '-0001.params')
+ files.append(model_name + '-symbol.json')
+ files.append(model_name + '-inference')
+
+ mxnet_folder = str(mxnet_version) + backslash + model_name + backslash
+
+ upload_model_files_to_s3(files, mxnet_folder)
+
+ clean_model_files(files, model_name)
+
+def train_lenet_gluon_save_params_api():
+ model_name = 'lenet_gluon_save_params_api'
+ print ('Saving files for model %s' %model_name)
+ net = Net()
+ weights = mx.initializer.Xavier(magnitude = 2.57)
+ net.initialize(weights, ctx = [mx.cpu(0)])
+ ### Prepare data
+
+ test_data = mx.nd.array(np.random.uniform(-1, 1, size=(20, 1, 30, 30)))
+ output = net(test_data)
+ # print (y)
+ #### Save the test data as well.
+ #### Save the inference output ys
+ #### Save the model params
+
+ mx.nd.save(model_name + '-data', {'data' : test_data})
+ save_inference_results(output, model_name)
+ net.save_params(model_name + '-params')
+
+ mxnet_folder = str(mxnet_version) + backslash + model_name + backslash
+
+ files = list()
+ files.append(model_name + '-data')
+ files.append(model_name + '-inference')
+ files.append(model_name + '-params')
+
+ upload_data_and_labels_to_s3(model_name)
+
+ upload_model_files_to_s3(files, mxnet_folder)
+
+ clean_model_files(files, model_name)
+
+def train_lenet_gluon_hybrid_export_api():
+ model_name = 'lenet_gluon_hybrid_export_api'
+ print ('Saving files for model %s' %model_name)
+ net = HybridNet()
+ weights = mx.initializer.Xavier(magnitude = 2.57)
+ net.initialize(weights, ctx = [mx.cpu(0)])
+ net.hybridize()
+ ### Prepare data
+ test_data = mx.nd.array(np.random.uniform(-1, 1, size=(20, 1, 30, 30)))
+ output = net(test_data)
+ # print (y)
+### Save the test data as well.
+### Save the inference output ys
+### Save the model params
+
+ mx.nd.save(model_name + '-data', {'data' : test_data})
+ save_inference_results(output, model_name)
+ net.export(model_name, epoch=1)
+
+ mxnet_folder = str(mxnet_version) + backslash + model_name + backslash
+
+ files = list()
+ files.append(model_name + '-data')
+ files.append(model_name + '-inference')
+ files.append(model_name + '-0001.params')
+ files.append(model_name + '-symbol.json')
+
+
+ upload_data_and_labels_to_s3(model_name)
+
+ upload_model_files_to_s3(files, mxnet_folder)
+
+ clean_model_files(files, model_name)
+
+def train_lstm_gluon_save_parameters_api():
+ ## If this code is being run on version >= 1.2.0 only then execute it,
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r201795191
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python mnist_mlp_module_api_train.py
Review comment:
Also, the training script would not be a part of nightly runs. It will only
be run when a new version of MXNet is released in order to refresh the trained
models inventory on S3.
This run can either be manual or automated by editing the
train_mxnet_legacy_models.sh script.
So any random dataset unavailability issues (ideally) should not block the
nightly inference files.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r201795191
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python mnist_mlp_module_api_train.py
Review comment:
Also, the training script would not be a part of nightly runs. It will only
be run when a new version of MXNet is released in order to refresh the trained
models inventory on S3.
This run can either be manual or automated by editing the
train_mxnet_legacy_models.sh script.
Plus I'm storing the datasets in S3 buckets, so I'm replying on the
availability of S3.
So any random dataset unavailability issues (ideally) should not block the
nightly inference files.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r201795191
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python mnist_mlp_module_api_train.py
Review comment:
Also, the training script would not be a part of nightly runs. It will only
be run when a new version of MXNet is released in order to refresh the trained
models inventory on S3.
This run can either be manual or automated by editing the
train_mxnet_legacy_models.sh script.
So any random dataset unavailability issues should not block the nightly
inference files.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r201791100
##
File path: tests/nightly/model_backwards_compatibility_check/common.py
##
@@ -165,3 +177,163 @@ def forward(self, x):
x = F.tanh(self.fc1(x))
x = F.tanh(self.fc2(x))
return x
+
+class Dictionary(object):
+def __init__(self):
+self.word2idx = {}
+self.idx2word = []
+
+def add_word(self, word):
+if word not in self.word2idx:
+self.idx2word.append(word)
+self.word2idx[word] = len(self.idx2word) - 1
+return self.word2idx[word]
+
+def __len__(self):
+return len(self.idx2word)
+
+class Corpus(object):
+def __init__(self, path):
+self.dictionary = Dictionary()
+self.download_data_from_s3()
+self.train = self.tokenize(path + 'train.txt')
+self.valid = self.tokenize(path + 'valid.txt')
+self.test = self.tokenize(path + 'test.txt')
+
+def download_data_from_s3(self, ):
+print ('Downloading files from bucket : ptb-small-dataset' )
+bucket = s3.Bucket('ptb-small-dataset')
Review comment:
Thanks. Replaced PTB with Sherlock in :
c3c91295b7c4e11241b5a32d4885c3d775f052b6
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r201791110
##
File path:
tests/nightly/model_backwards_compatibility_check/lm_rnn_gluon_inference.py
##
@@ -17,197 +17,31 @@
# specific language governing permissions and limitations
# under the License.
-import math
-import os
-import time
-import numpy as np
-import mxnet as mx
-from mxnet import gluon, autograd
-from mxnet.gluon import nn, rnn
-import logging
-import boto3
-import json
-logging.getLogger().setLevel(logging.DEBUG)
-mx.random.seed(7)
-np.random.seed(7)
+from common import *
-mxnet_version = mx.__version__
-bucket_name = 'mxnet-model-backwards-compatibility'
-ctx = mx.cpu()
-backslash = '/'
model_name = 'lm_rnn_gluon_api'
-s3 = boto3.resource('s3')
-
-
-args_data = 'ptb.'
-args_model = 'rnn_relu'
-args_emsize = 100
-args_nhid = 100
-args_nlayers = 2
-args_lr = 1.0
-args_clip = 0.2
-args_epochs = 2
-args_batch_size = 32
-args_bptt = 5
-args_dropout = 0.2
-args_tied = True
-args_cuda = 'store_true'
-args_log_interval = 500
-args_save = model_name + '.params'
-
-class Dictionary(object):
-def __init__(self):
-self.word2idx = {}
-self.idx2word = []
-
-def add_word(self, word):
-if word not in self.word2idx:
-self.idx2word.append(word)
-self.word2idx[word] = len(self.idx2word) - 1
-return self.word2idx[word]
-
-def __len__(self):
-return len(self.idx2word)
-
-class Corpus(object):
-def __init__(self, path):
-self.dictionary = Dictionary()
-self.download_data_from_s3()
-self.train = self.tokenize(path + 'train.txt')
-self.valid = self.tokenize(path + 'valid.txt')
-self.test = self.tokenize(path + 'test.txt')
-
-def download_data_from_s3(self):
- print ('Downloading files from bucket : %s' %bucket_name)
- bucket = s3.Bucket(bucket_name)
- files = ['test.txt', 'train.txt', 'valid.txt']
- for file in files:
- if os.path.exists(args_data + file) :
- print ('File %s'%(args_data + file), 'already exists.
Skipping download')
- continue
- file_path = str(mxnet_version) + backslash + model_name +
backslash + args_data + file
- bucket.download_file(file_path, args_data + file)
-
-def tokenize(self, path):
-"""Tokenizes a text file."""
-assert os.path.exists(path)
-# Add words to the dictionary
-with open(path, 'r') as f:
-tokens = 0
-for line in f:
-words = line.split() + ['']
-tokens += len(words)
-for word in words:
-self.dictionary.add_word(word)
-
-# Tokenize file content
-with open(path, 'r') as f:
-ids = np.zeros((tokens,), dtype='int32')
-token = 0
-for line in f:
-words = line.split() + ['']
-for word in words:
-ids[token] = self.dictionary.word2idx[word]
-token += 1
-
-return mx.nd.array(ids, dtype='int32')
-
-class RNNModel(gluon.Block):
-"""A model with an encoder, recurrent layer, and a decoder."""
-
-def __init__(self, mode, vocab_size, num_embed, num_hidden,
- num_layers, dropout=0.5, tie_weights=False, **kwargs):
-super(RNNModel, self).__init__(**kwargs)
-with self.name_scope():
-self.drop = nn.Dropout(dropout)
-self.encoder = nn.Embedding(vocab_size, num_embed,
-weight_initializer =
mx.init.Uniform(0.1))
-if mode == 'rnn_relu':
-self.rnn = rnn.RNN(num_hidden, num_layers, activation='relu',
dropout=dropout,
- input_size=num_embed)
-elif mode == 'rnn_tanh':
-self.rnn = rnn.RNN(num_hidden, num_layers, dropout=dropout,
- input_size=num_embed)
-elif mode == 'lstm':
-self.rnn = rnn.LSTM(num_hidden, num_layers, dropout=dropout,
-input_size=num_embed)
-elif mode == 'gru':
-self.rnn = rnn.GRU(num_hidden, num_layers, dropout=dropout,
- input_size=num_embed)
-else:
-raise ValueError("Invalid mode %s. Options are rnn_relu, "
- "rnn_tanh, lstm, and gru"%mode)
-if tie_weights:
-self.decoder = nn.Dense(vocab_size, in_units = num_hidden,
-params = self.encoder.params)
-else:
-self.decoder = nn.Dense(vocab_size, in_units = num_hidden)
-self.num_hidden = num_hidden
-
-def forward(self,
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r201782584
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python mnist_mlp_module_api_train.py
Review comment:
That's gracefully handled in this commit :
https://github.com/apache/incubator-mxnet/pull/11626/commits/7c41488fe3d8679ec12a5d0694a596cc6a72017d
As of now, I'm exiting the inference script with a message indicating no
models found in S3.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r201780568
##
File path:
tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh
##
@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#Author: Piyush Ghai
+
+run_models() {
+ echo '=='
+ echo "Running training files and preparing models"
+ echo '=='
+ python mnist_mlp_module_api_train.py
Review comment:
I did not quite fully understand by what you meant by integrity here.
Are you suggesting that I do a checksum validation on the generated models
and other relevant files while downloading them in the inference files ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet Model Backwards Compatibility Checker
piyushghai commented on a change in pull request #11626: [MXNET-651] MXNet
Model Backwards Compatibility Checker
URL: https://github.com/apache/incubator-mxnet/pull/11626#discussion_r201780155
##
File path: tests/nightly/model_backwards_compatibility_check/common.py
##
@@ -165,3 +177,163 @@ def forward(self, x):
x = F.tanh(self.fc1(x))
x = F.tanh(self.fc2(x))
return x
+
+class Dictionary(object):
+def __init__(self):
+self.word2idx = {}
+self.idx2word = []
+
+def add_word(self, word):
+if word not in self.word2idx:
+self.idx2word.append(word)
+self.word2idx[word] = len(self.idx2word) - 1
+return self.word2idx[word]
+
+def __len__(self):
+return len(self.idx2word)
+
+class Corpus(object):
+def __init__(self, path):
+self.dictionary = Dictionary()
+self.download_data_from_s3()
+self.train = self.tokenize(path + 'train.txt')
+self.valid = self.tokenize(path + 'valid.txt')
+self.test = self.tokenize(path + 'test.txt')
+
+def download_data_from_s3(self, ):
+print ('Downloading files from bucket : ptb-small-dataset' )
+bucket = s3.Bucket('ptb-small-dataset')
Review comment:
Oh. I wasn't aware of the Penn Tree Bank dataset license issue. I had
uploaded a trimmed down version of PTB to an S3 bucket. The trimmed down
version was 10k training sentences and 3k each for test and validation.
Can you suggest a different dataset which we can host ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
