sandeep-krishnamurthy closed pull request #12490: Remove broken link regression test, fix broken links URL: https://github.com/apache/incubator-mxnet/pull/12490
This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/architecture/rnn_interface.md b/docs/architecture/rnn_interface.md index 42338763ce6..dc0b6a7958e 100644 --- a/docs/architecture/rnn_interface.md +++ b/docs/architecture/rnn_interface.md @@ -1,6 +1,6 @@ # Survey of Existing Interfaces and Implementations -Commonly used deep learning libraries with good RNN/LSTM support include [Theano](http://deeplearning.net/software/theano/library/scan.html) and its wrappers [Lasagne](http://lasagne.readthedocs.org/en/latest/modules/layers/recurrent.html) and [Keras](http://keras.io/layers/recurrent/); [CNTK](https://cntk.codeplex.com/); [TensorFlow](https://www.tensorflow.org/versions/master/tutorials/recurrent/index.html); and various implementations in Torch, such as [this well-known character-level language model tutorial](https://github.com/karpathy/char-rnn), [this](https://github.com/Element-Research/rnn). +Commonly used deep learning libraries with good RNN/LSTM support include [Theano](http://deeplearning.net/software/theano/library/scan.html) and its wrappers [Lasagne](http://lasagne.readthedocs.org/en/latest/modules/layers/recurrent.html) and [Keras](http://keras.io/layers/recurrent/); [CNTK](https://cntk.codeplex.com/); [TensorFlow](https://www.tensorflow.org/tutorials/sequences/recurrent); and various implementations in Torch, such as [this well-known character-level language model tutorial](https://github.com/karpathy/char-rnn), [this](https://github.com/Element-Research/rnn). In this document, we present a comparative analysis of the approaches taken by these libraries. @@ -93,7 +93,7 @@ The low-level API for recurrent connection seem to be a *delay node*. But I'm no ## TensorFlow -The [current example of RNNLM](https://www.tensorflow.org/versions/master/tutorials/recurrent/index.html#recurrent-neural-networks) in TensorFlow uses explicit unrolling for a predefined number of time steps. The white-paper mentions that an advanced control flow API (Theano's scan-like) is planned. +The [current example of RNNLM](https://www.tensorflow.org/tutorials/sequences/recurrent#recurrent-neural-networks) in TensorFlow uses explicit unrolling for a predefined number of time steps. The white-paper mentions that an advanced control flow API (Theano's scan-like) is planned. ## Next Steps diff --git a/docs/install/index.md b/docs/install/index.md index 4a6af31cee3..3a697ae20ee 100644 --- a/docs/install/index.md +++ b/docs/install/index.md @@ -272,7 +272,7 @@ Follow the four steps in this [docker documentation](https://docs.docker.com/eng If you skip this step, you need to use *sudo* each time you invoke Docker. -**Step 3** Install *nvidia-docker-plugin* following the [installation instructions](https://github.com/NVIDIA/nvidia-docker/wiki/Installation). *nvidia-docker-plugin* is required to enable the usage of GPUs from the docker containers. +**Step 3** Install *nvidia-docker-plugin* following the [installation instructions](https://github.com/NVIDIA/nvidia-docker/wiki). *nvidia-docker-plugin* is required to enable the usage of GPUs from the docker containers. **Step 4** Pull the MXNet docker image. diff --git a/docs/install/windows_setup.md b/docs/install/windows_setup.md index 99ce7f63e85..c974eeb858b 100755 --- a/docs/install/windows_setup.md +++ b/docs/install/windows_setup.md @@ -55,7 +55,7 @@ These commands produce a library called ```mxnet.dll``` in the ```./build/Releas Next, we install ```graphviz``` library that we use for visualizing network graphs you build on MXNet. We will also install [Jupyter Notebook](http://jupyter.readthedocs.io/) used for running MXNet tutorials and examples. - Install ```graphviz``` by downloading MSI installer from [Graphviz Download Page](https://graphviz.gitlab.io/_pages/Download/Download_windows.html). **Note** Make sure to add graphviz executable path to PATH environment variable. Refer [here for more details](http://stackoverflow.com/questions/35064304/runtimeerror-make-sure-the-graphviz-executables-are-on-your-systems-path-aft) -- Install ```Jupyter``` by installing [Anaconda for Python 2.7](https://www.continuum.io/downloads) +- Install ```Jupyter``` by installing [Anaconda for Python 2.7](https://www.anaconda.com/download/) **Note** Do not install Anaconda for Python 3.5. MXNet has few compatibility issue with Python 3.5. @@ -69,7 +69,7 @@ We have installed MXNet core library. Next, we will install MXNet interface pack ## Install MXNet for Python 1. Install ```Python``` using windows installer available [here](https://www.python.org/downloads/release/python-2712/). -2. Install ```Numpy``` using windows installer available [here](http://scipy.org/install.html). +2. Install ```Numpy``` using windows installer available [here](https://scipy.org/index.html). 3. Next, we install Python package interface for MXNet. You can find the Python interface package for [MXNet on GitHub](https://github.com/dmlc/mxnet/tree/master/python/mxnet). ```bash diff --git a/docs/tutorials/onnx/export_mxnet_to_onnx.md b/docs/tutorials/onnx/export_mxnet_to_onnx.md index a9c03bed8b1..dc34bd520b4 100644 --- a/docs/tutorials/onnx/export_mxnet_to_onnx.md +++ b/docs/tutorials/onnx/export_mxnet_to_onnx.md @@ -55,7 +55,7 @@ Help on function export_model in module mxnet.contrib.onnx.mx2onnx.export_model: export_model(sym, params, input_shape, input_type=<type 'numpy.float32'>, onnx_file_path=u'model.onnx', verbose=False) Exports the MXNet model file, passed as a parameter, into ONNX model. Accepts both symbol,parameter objects as well as json and params filepaths as input. - Operator support and coverage - https://cwiki.apache.org/confluence/display/MXNET/ONNX + Operator support and coverage - https://cwiki.apache.org/confluence/display/MXNET/MXNet-ONNX+Integration Parameters ---------- diff --git a/python/mxnet/contrib/onnx/mx2onnx/export_model.py b/python/mxnet/contrib/onnx/mx2onnx/export_model.py index 33292bf664a..e5158051d6f 100644 --- a/python/mxnet/contrib/onnx/mx2onnx/export_model.py +++ b/python/mxnet/contrib/onnx/mx2onnx/export_model.py @@ -36,7 +36,8 @@ def export_model(sym, params, input_shape, input_type=np.float32, onnx_file_path='model.onnx', verbose=False): """Exports the MXNet model file, passed as a parameter, into ONNX model. Accepts both symbol,parameter objects as well as json and params filepaths as input. - Operator support and coverage - https://cwiki.apache.org/confluence/display/MXNET/ONNX + Operator support and coverage - + https://cwiki.apache.org/confluence/display/MXNET/MXNet-ONNX+Integration Parameters ---------- diff --git a/python/mxnet/contrib/onnx/onnx2mx/import_model.py b/python/mxnet/contrib/onnx/onnx2mx/import_model.py index e190c3bdadc..b8d3bf28ee2 100644 --- a/python/mxnet/contrib/onnx/onnx2mx/import_model.py +++ b/python/mxnet/contrib/onnx/onnx2mx/import_model.py @@ -23,7 +23,8 @@ def import_model(model_file): """Imports the ONNX model file, passed as a parameter, into MXNet symbol and parameters. - Operator support and coverage - https://cwiki.apache.org/confluence/display/MXNET/ONNX + Operator support and coverage - + https://cwiki.apache.org/confluence/display/MXNET/MXNet-ONNX+Integration Parameters ---------- diff --git a/python/mxnet/contrib/text/embedding.py b/python/mxnet/contrib/text/embedding.py index 38defb4b90b..277f7822292 100644 --- a/python/mxnet/contrib/text/embedding.py +++ b/python/mxnet/contrib/text/embedding.py @@ -490,7 +490,7 @@ class GloVe(_TokenEmbedding): License for pre-trained embeddings: - https://opendatacommons.org/licenses/pddl/ + https://fedoraproject.org/wiki/Licensing/PDDL Parameters diff --git a/tests/nightly/broken_link_checker_test/JenkinsfileForBLC b/tests/nightly/broken_link_checker_test/JenkinsfileForBLC index 782bf74c9cc..4c3f0531997 100755 --- a/tests/nightly/broken_link_checker_test/JenkinsfileForBLC +++ b/tests/nightly/broken_link_checker_test/JenkinsfileForBLC @@ -34,11 +34,7 @@ core_logic: { timeout(time: 60, unit: 'MINUTES') { try { utils.init_git() - sh 'aws s3 cp s3://mxnet-ci-prod-slave-data/url_list.txt ./tests/nightly/broken_link_checker_test/url_list.txt' utils.docker_run('ubuntu_blc', 'broken_link_checker', false) - } finally { - sh "echo Storing the new url_list.txt to S3 bucket" - sh 'aws s3 cp ./tests/nightly/broken_link_checker_test/url_list.txt s3://mxnet-ci-prod-slave-data/url_list.txt' } } } diff --git a/tests/nightly/broken_link_checker_test/README.md b/tests/nightly/broken_link_checker_test/README.md index a925d1b4381..c39abd0d617 100755 --- a/tests/nightly/broken_link_checker_test/README.md +++ b/tests/nightly/broken_link_checker_test/README.md @@ -1,6 +1,6 @@ # Broken link checker test -This folder contains the scripts that are required to run the nightly job of checking the broken links. The job also checks whether the link that were published before are still accessible. +This folder contains the scripts that are required to run the nightly job of checking the broken links. ## JenkinsfileForBLC This is configuration file for jenkins job. @@ -8,6 +8,3 @@ This is configuration file for jenkins job. ## Details The `broken_link_checker.sh` is a top level script that invokes the `test_broken_links.py` and `check_regression.sh` scripts. The `test_broken_links.py` invokes broken link checker tool (blc) from nodeJs and reports the list of URLs that are not accessible. -The `check_regression.sh` scripts downloads the file `url_list.txt` that contains links that are publicly accessible from s3 bucket -The scripts merges this list with the output of `test_broken_links.py` and checks whether all those links are accessible using 'curl' command. -The updated `url_list.txt` is uploaded to s3 bucket. diff --git a/tests/nightly/broken_link_checker_test/broken_link_checker.sh b/tests/nightly/broken_link_checker_test/broken_link_checker.sh index 2107c96d257..450cd65b8d9 100755 --- a/tests/nightly/broken_link_checker_test/broken_link_checker.sh +++ b/tests/nightly/broken_link_checker_test/broken_link_checker.sh @@ -28,6 +28,3 @@ echo `pwd` echo "Running test_broken_links.py" python test_broken_links.py - -echo "Running check_regression.sh" -./check_regression.sh diff --git a/tests/nightly/broken_link_checker_test/check_regression.sh b/tests/nightly/broken_link_checker_test/check_regression.sh deleted file mode 100755 index c21577f3ab3..00000000000 --- a/tests/nightly/broken_link_checker_test/check_regression.sh +++ /dev/null @@ -1,46 +0,0 @@ -#! /bin/sh - -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - - -echo "Running the check_regression.sh script" -cat blc_output.txt | uniq | grep -Eo "(http|https).* " | sort| uniq > unique_current_urls.txt - -cat url_list.txt unique_current_urls.txt | sort | uniq > new_url_list.txt -regression=false -while IFS= read -r line -do - err=$(curl -Is $line | head -n 1 | grep 404) - if [ "$err" ]; then - if [ "$regression" = false ] ; then - echo "FAIL: REGRESSION" - regression=true - fi - echo "BROKEN $line $err" - fi - unset err -done < new_url_list.txt -mv new_url_list.txt url_list.txt -rm -rf unique_current_urls.txt -rm -rf blc_output.txt -if [ $regression ]; then - echo "FAIL: Found Regression in broken link checker" - exit 1 -else - echo "SUCCESS: No Regression found" -fi diff --git a/tests/nightly/broken_link_checker_test/test_broken_links.py b/tests/nightly/broken_link_checker_test/test_broken_links.py index 593e008d508..d1faf497519 100755 --- a/tests/nightly/broken_link_checker_test/test_broken_links.py +++ b/tests/nightly/broken_link_checker_test/test_broken_links.py @@ -31,6 +31,7 @@ def prepare_link_test_result(command_output): # Whitelisted broken links patterns HTTP_403_REGEX = "(HTTP_403)" HTTP_401_REGEX = "(HTTP_401)" + HTTP_409_REGEX = "(HTTP_409)" BLC_UNKNOWN_REGEX = "(BLC_UNKNOWN)" HTTP_UNDEFINED = "HTTP_undefined" FALSE_SCALA_API_DOC_LINK = "java$lang.html" @@ -53,8 +54,8 @@ def prepare_link_test_result(command_output): current_page_broken_links = "" if line.find(BROKEN_PAGE_START_REGEX) != -1: - # Skip (401, 403, unknown issues) - if HTTP_403_REGEX not in line and HTTP_401_REGEX not in line and BLC_UNKNOWN_REGEX not in line and HTTP_UNDEFINED not in line and FALSE_SCALA_API_DOC_LINK not in line and FALSE_SCALA_API_DEPRECATED_LINK not in line and FALSE_PAPER_LINK not in line: + # Skip (401, 403, 409, unknown issues) + if HTTP_403_REGEX not in line and HTTP_401_REGEX not in line and HTTP_409_REGEX not in line and BLC_UNKNOWN_REGEX not in line and HTTP_UNDEFINED not in line and FALSE_SCALA_API_DOC_LINK not in line and FALSE_SCALA_API_DEPRECATED_LINK not in line and FALSE_PAPER_LINK not in line: current_page_broken = True current_page_broken_links += line.split(BROKEN_PAGE_START_REGEX)[1] + "\n" ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
