indhub closed pull request #10495: [MXNET-307] Add tutorials to the CI + Fix them URL: https://github.com/apache/incubator-mxnet/pull/10495
This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/.gitignore b/.gitignore index 6be98c50466..d585672ab7d 100644 --- a/.gitignore +++ b/.gitignore @@ -136,6 +136,7 @@ lib #Notebook Automated Test !tests/nightly/test_tutorial_config.txt !tests/nightly/TestNotebook +tests/nightly/tmp_notebook # pip building tools tools/pip_package/build diff --git a/docs/tutorials/basic/data.md b/docs/tutorials/basic/data.md index 54ee334f978..fdd50150f33 100644 --- a/docs/tutorials/basic/data.md +++ b/docs/tutorials/basic/data.md @@ -15,12 +15,6 @@ To complete this tutorial, we need: ``` $ pip install opencv-python requests matplotlib jupyter ``` -- Set the environment variable `MXNET_HOME` to the root of the MXNet source folder. - -``` -$ git clone https://github.com/dmlc/mxnet ~/mxnet -$ export MXNET_HOME='~/mxnet' -``` ## MXNet Data Iterator Data Iterators in *MXNet* are similar to Python iterator objects. @@ -61,6 +55,11 @@ we can use the [__`NDArrayIter`__](http://mxnet.io/api/python/io/io.html#mxnet.i ```python import numpy as np + +# fix the seed +np.random.seed(42) +mx.random.seed(42) + data = np.random.rand(100,3) label = np.random.randint(0, 10, (100,)) data_iter = mx.io.NDArrayIter(data=data, label=label, batch_size=30) @@ -341,7 +340,7 @@ Let's download sample images that we can work with. ```python fname = mx.test_utils.download(url='http://data.mxnet.io/data/test_images.tar.gz', dirname='data', overwrite=False) tar = tarfile.open(fname) -tar.extractall(path='./data') +tar.extractall(path=os.path.join('.','data')) tar.close() ``` @@ -351,7 +350,7 @@ tar.close() **Note:** You will still need ``OpenCV``(not the CV2 Python library) installed to use `mx.image.imdecode`. ```python -img = mx.image.imdecode(open('data/test_images/ILSVRC2012_val_00000001.JPEG', 'rb').read()) +img = mx.image.imdecode(open(os.path.join('data','test_images','ILSVRC2012_val_00000001.JPEG'), 'rb').read()) plt.imshow(img.asnumpy()); plt.show() ``` @@ -382,7 +381,7 @@ Download and unzip ```python fname = mx.test_utils.download(url='http://www.vision.caltech.edu/Image_Datasets/Caltech101/101_ObjectCategories.tar.gz', dirname='data', overwrite=False) tar = tarfile.open(fname) -tar.extractall(path='./data') +tar.extractall(path=os.path.join('.','data')) tar.close() ``` @@ -392,7 +391,14 @@ Now let's convert them into record io format using the `im2rec.py` utility scrip First, we need to make a list that contains all the image files and their categories: ```python -os.system('python %s/tools/im2rec.py --list --recursive --test-ratio=0.2 data/caltech data/101_ObjectCategories'%os.environ['MXNET_HOME']) +mxnet_path = os.path.dirname(mx.__file__) +im2rec_path = os.path.join(mxnet_path, 'tools','im2rec.py') +data_path = os.path.join('data','101_ObjectCategories') +prefix_path = os.path.join('data','caltech') + +with open(os.devnull, 'wb') as devnull: + subprocess.check_call(['python', im2rec_path, '--list', '--recursive', '--test-ratio=0.2', prefix_path, data_path], + stdout=devnull) ``` The resulting list file (./data/caltech_train.lst) is in the format `index\t(one or more label)\tpath`. In this case, there is only one label for each image but you can modify the list to add in more for multi-label training. @@ -401,7 +407,9 @@ Then we can use this list to create our record io file: ```python -os.system("python %s/tools/im2rec.py --num-thread=4 --pass-through data/caltech data/101_ObjectCategories"%os.environ['MXNET_HOME']) +with open(os.devnull, 'wb') as devnull: + subprocess.check_call(['python', im2rec_path, '--num-thread=4', '--pass-through', '--test-ratio=0.2', prefix_path, data_path], + stdout=devnull) ``` The record io files are now saved at here (./data) @@ -412,7 +420,7 @@ The record io files are now saved at here (./data) ```python data_iter = mx.io.ImageRecordIter( - path_imgrec="./data/caltech.rec", # the target record file + path_imgrec=os.path.join('.','data','caltech.rec'), data_shape=(3, 227, 227), # output data shape. An 227x227 region will be cropped from the original image. batch_size=4, # number of samples per batch resize=256 # resize the shorter edge to 256 before cropping @@ -433,8 +441,8 @@ plt.show() ```python data_iter = mx.image.ImageIter(batch_size=4, data_shape=(3, 227, 227), - path_imgrec="./data/caltech.rec", - path_imgidx="./data/caltech.idx" ) + path_imgrec=os.path.join('.','data','caltech.rec'), + path_imgidx=os.path.join('.','data','caltech.idx') ) data_iter.reset() batch = data_iter.next() data = batch.data[0] diff --git a/docs/tutorials/basic/image_io.md b/docs/tutorials/basic/image_io.md deleted file mode 100644 index 092affbc74c..00000000000 --- a/docs/tutorials/basic/image_io.md +++ /dev/null @@ -1,99 +0,0 @@ -# Image IO - Loading and pre-processing images - -This tutorial explains how to prepare, load and train with image data in -MXNet. All IO in MXNet is handled via `mx.io.DataIter` and its subclasses. In -this tutorial we focus on how to use pre-built data iterators as while as custom -iterators to process image data. - -There are mainly three ways of loading image data in MXNet: - -- [NEW] [mx.img.ImageIter](https://mxnet.incubator.apache.org/versions/master/api/python/image/image.html#mxnet.image.ImageIter): implemented in python, easily customizable, can load - from both .rec files and raw image files. -- [OLD] [mx.io.ImageRecordIter](https://mxnet.incubator.apache.org/versions/master/api/python/io.html#mxnet.io.ImageRecordIter): implemented in backend (C++), less customizable - but can be used in all language bindings, load from .rec files -- Custom iterator by inheriting mx.io.DataIter - -First, we explain the record io file format used by mxnet: - -## RecordIO - -[Record IO](https://mxnet.incubator.apache.org/architecture/note_data_loading.html#data-format) is the main file format used by MXNet for data IO. It supports reading -and writing on various file systems including distributed file systems like -Hadoop HDFS and AWS S3. First, we download the Caltech 101 dataset that -contains 101 classes of objects and convert them into record io format: - -```python -%matplotlib inline -import os -import subprocess -import mxnet as mx -import numpy as np -import matplotlib.pyplot as plt - -# change this to your mxnet location -MXNET_HOME = '/scratch/mxnet' -``` - -Download and unzip the dataset. The dataset is about ~126MB and may take some time: - -```python -os.system('wget http://www.vision.caltech.edu/Image_Datasets/Caltech101/101_ObjectCategories.tar.gz -P data/') -os.chdir('data') -os.system('tar -xf 101_ObjectCategories.tar.gz') -os.chdir('../') -``` - -Let's take a look at the data. - -As you can see, under the -root folder (data/101_ObjectCategories) every category has a -subfolder (e.g. data/101_ObjectCategories/yin_yang). - -Now let's convert them into record io format. First we need to make a list that -contains all the image files and their categories: - - -```python -assert(MXNET_HOME != '/scratch/mxnet'), "Please update your MXNet location" -os.system('python %s/tools/im2rec.py --list --recursive --test-ratio=0.2 data/caltech data/101_ObjectCategories'%MXNET_HOME) -``` - -The resulting [list file](./data/caltech_train.lst) is in the format -`index\t(one or more label)\tpath`. In this case there is only one label for -each image but you can modify the list to add in more for multi label training. - -Then we can use this list to create our record io file: - - -```python -os.system("python %s/tools/im2rec.py --num-thread=4 --pass-through data/caltech data/101_ObjectCategories"%MXNET_HOME) -``` - -The record io files are now saved in the "data" directory. - -## ImageRecordIter - -`mx.io.ImageRecordIter` can be used for loading image data saved in record io -format. It is available in all frontend languages, but as it's implemented in -C++, it is less flexible. - -To use ImageRecordIter, simply create an instance by loading your record file: - -```python -data_iter = mx.io.ImageRecordIter( - path_imgrec="./data/caltech_train.rec", # the target record file - data_shape=(3, 227, 227), # output data shape. An 227x227 region will be cropped from the original image. - batch_size=4, # number of samples per batch - resize=256 # resize the shorter edge to 256 before cropping - # ... you can add more augmentation options here. use help(mx.io.ImageRecordIter) to see all possible choices - ) -data_iter.reset() -batch = data_iter.next() -data = batch.data[0] -for i in range(4): - plt.subplot(1,4,i+1) - plt.imshow(data[i].asnumpy().astype(np.uint8).transpose((1,2,0))) -plt.show() -``` - -<!-- INSERT SOURCE DOWNLOAD BUTTONS --> diff --git a/docs/tutorials/basic/ndarray_indexing.md b/docs/tutorials/basic/ndarray_indexing.md index 37168b34014..35dd8c17f67 100644 --- a/docs/tutorials/basic/ndarray_indexing.md +++ b/docs/tutorials/basic/ndarray_indexing.md @@ -15,7 +15,7 @@ In MXNet, we support both basic and advanced indexing following the convention o ## Basic Slicing and Indexing -Basic slicing extends Python’s basic concept of slicing to N dimensions. For a quick review: +Basic slicing extends Python's basic concept of slicing to N dimensions. For a quick review: ``` a[start:end] # items start through end-1 @@ -263,7 +263,7 @@ Advanced indexing always returns a __copy__ of the data. When the index consists of as many integer arrays as the array being indexed has dimensions, the indexing is straight forward, but different from slicing. Advanced indexes always are [broadcast](https://docs.scipy.org/doc/numpy-1.13.0/reference/ufuncs.html#ufuncs-broadcasting) and iterated as one: -```python +``` result[i_1, ..., i_M] == x[ind_1[i_1, ..., i_M], ind_2[i_1, ..., i_M], ..., ind_N[i_1, ..., i_M]] ``` diff --git a/docs/tutorials/basic/record_io.md b/docs/tutorials/basic/record_io.md deleted file mode 100644 index 9ba6fa6e25f..00000000000 --- a/docs/tutorials/basic/record_io.md +++ /dev/null @@ -1,129 +0,0 @@ -# Record IO - Pack free-format data in binary files - -This tutorial will walk through the python interface for reading and writing -record io files. It can be useful when you need more more control over the -details of data pipeline. For example, when you need to augment image and label -together for detection and segmentation, or when you need a custom data iterator -for triplet sampling and negative sampling. - -Setup environment first: - -```python -%matplotlib inline -from __future__ import print_function -import mxnet as mx -import numpy as np -import matplotlib.pyplot as plt -``` - -The relevant code is under `mx.recordio`. There are two classes: `MXRecordIO`, -which supports sequential read and write, and `MXIndexedRecordIO`, which -supports random read and sequential write. - -## MXRecordIO - -First let's take a look at `MXRecordIO`. We open a file `tmp.rec` and write 5 -strings to it: - -```python -record = mx.recordio.MXRecordIO('tmp.rec', 'w') -for i in range(5): - record.write('record_%d'%i) -record.close() -``` - -Then we can read it back by opening the same file with 'r': - -```python -record = mx.recordio.MXRecordIO('tmp.rec', 'r') -while True: - item = record.read() - if not item: - break - print item -record.close() -``` - -## MXIndexedRecordIO - -Some times you need random access for more complex tasks. `MXIndexedRecordIO` is -designed for this. Here we create a indexed record `tmp.rec` and a corresponding -index file `tmp.idx`: - - -```python -record = mx.recordio.MXIndexedRecordIO('tmp.idx', 'tmp.rec', 'w') -for i in range(5): - record.write_idx(i, 'record_%d'%i) -record.close() -``` - -We can then access records with keys: - -```python -record = mx.recordio.MXIndexedRecordIO('tmp.idx', 'tmp.rec', 'r') -record.read_idx(3) -``` - -You can list all keys with: - -```python -record.keys -``` - -## Packing and Unpacking Data - -Each record in a .rec file can contain arbitrary binary data, but machine -learning data typically has a label/data structure. `mx.recordio` also contains -a few utility functions for packing such data, namely: `pack`, `unpack`, -`pack_img`, and `unpack_img`. - -### Binary Data - -`pack` and `unpack` are used for storing float (or 1d array of float) label and -binary data: - -- pack: - -```python -# pack -data = 'data' -label1 = 1.0 -header1 = mx.recordio.IRHeader(flag=0, label=label1, id=1, id2=0) -s1 = mx.recordio.pack(header1, data) -print('float label:', repr(s1)) -label2 = [1.0, 2.0, 3.0] -header2 = mx.recordio.IRHeader(flag=0, label=label2, id=2, id2=0) -s2 = mx.recordio.pack(header2, data) -print('array label:', repr(s2)) -``` - -- unpack: - -```python -print(*mx.recordio.unpack(s1)) -print(*mx.recordio.unpack(s2)) -``` - -### Image Data - -`pack_img` and `unpack_img` are used for packing image data. Records packed by -`pack_img` can be loaded by `mx.io.ImageRecordIter`. - -- pack images - -```python -data = np.ones((3,3,1), dtype=np.uint8) -label = 1.0 -header = mx.recordio.IRHeader(flag=0, label=label, id=0, id2=0) -s = mx.recordio.pack_img(header, data, quality=100, img_fmt='.jpg') -print(repr(s)) -``` - -- unpack images - -```python -print(*mx.recordio.unpack_img(s)) -``` - -<!-- INSERT SOURCE DOWNLOAD BUTTONS --> diff --git a/docs/tutorials/gluon/hybrid.md b/docs/tutorials/gluon/hybrid.md index 6b4c6aafbba..859ad934c7e 100644 --- a/docs/tutorials/gluon/hybrid.md +++ b/docs/tutorials/gluon/hybrid.md @@ -36,6 +36,8 @@ import mxnet as mx from mxnet import gluon from mxnet.gluon import nn +mx.random.seed(42) + class Net(gluon.HybridBlock): def __init__(self, **kwargs): super(Net, self).__init__(**kwargs) @@ -43,9 +45,9 @@ class Net(gluon.HybridBlock): # layers created in name_scope will inherit name space # from parent layer. self.conv1 = nn.Conv2D(6, kernel_size=5) - self.pool1 = nn.Pool2D(kernel_size=2) + self.pool1 = nn.MaxPool2D(pool_size=2) self.conv2 = nn.Conv2D(16, kernel_size=5) - self.pool2 = nn.Pool2D(kernel_size=2) + self.pool2 = nn.MaxPool2D(pool_size=2) self.fc1 = nn.Dense(120) self.fc2 = nn.Dense(84) # You can use a Dense layer for fc3 but we do dot product manually diff --git a/docs/tutorials/gluon/mnist.md b/docs/tutorials/gluon/mnist.md index fc2271999fd..86c493b38fe 100644 --- a/docs/tutorials/gluon/mnist.md +++ b/docs/tutorials/gluon/mnist.md @@ -32,6 +32,10 @@ The following source code downloads and loads the images and the corresponding l ```python import mxnet as mx + +# Fixing the random seed +mx.random.seed(42) + mnist = mx.test_utils.get_mnist() ``` @@ -89,16 +93,17 @@ with net.name_scope(): The following source code initializes all parameters received from parameter dict using [Xavier](http://mxnet.io/api/python/optimization.html#mxnet.initializer.Xavier) initializer to train the MLP network we defined above. -For our training, we will make use of the stochastic gradient descent (SGD) optimizer. In particular, we'll be using mini-batch SGD. Standard SGD processes train data one example at a time. In practice, this is very slow and one can speed up the process by processing examples in small batches. In this case, our batch size will be 100, which is a reasonable choice. Another parameter we select here is the learning rate, which controls the step size the optimizer takes in search of a solution. We'll pick a learning rate of 0.1, again a reasonable choice. Settings such as batch size and learning rate are what are usually referred to as hyper-parameters. What values we give them can have a great impact on training performance. +For our training, we will make use of the stochastic gradient descent (SGD) optimizer. In particular, we'll be using mini-batch SGD. Standard SGD processes train data one example at a time. In practice, this is very slow and one can speed up the process by processing examples in small batches. In this case, our batch size will be 100, which is a reasonable choice. Another parameter we select here is the learning rate, which controls the step size the optimizer takes in search of a solution. We'll pick a learning rate of 0.02, again a reasonable choice. Settings such as batch size and learning rate are what are usually referred to as hyper-parameters. What values we give them can have a great impact on training performance. We will use [Trainer](http://mxnet.io/api/python/gluon.html#trainer) class to apply the [SGD optimizer](http://mxnet.io/api/python/optimization.html#mxnet.optimizer.SGD) on the initialized parameters. ```python -ctx = [mx.cpu(0), mx.cpu(1)] +gpus = mx.test_utils.list_gpus() +ctx = [mx.gpu()] if gpus else [mx.cpu(0), mx.cpu(1)] net.collect_params().initialize(mx.init.Xavier(magnitude=2.24), ctx=ctx) -trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.1}) +trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.02}) ``` #### Train the network @@ -120,10 +125,11 @@ There are many predefined loss functions in gluon.loss. Here we use training scope which is defined by `autograd.record()`. ```python +%%time epoch = 10 # Use Accuracy as the evaluation metric. metric = mx.metric.Accuracy() - +softmax_cross_entropy_loss = gluon.loss.SoftmaxCrossEntropyLoss() for i in range(epoch): # Reset the train data iterator. train_data.reset() @@ -141,9 +147,9 @@ for i in range(epoch): for x, y in zip(data, label): z = net(x) # Computes softmax cross entropy loss. - loss = gluon.loss.softmax_cross_entropy_loss(z, y) + loss = softmax_cross_entropy_loss(z, y) # Backpropagate the error for one iteration. - ag.backward([loss]) + loss.backward() outputs.append(z) # Updates internal evaluation metric.update(label, outputs) @@ -180,7 +186,7 @@ for batch in val_data: # Updates internal evaluation metric.update(label, outputs) print('validation acc: %s=%f'%metric.get()) -assert metric.get()[1] > 0.96 +assert metric.get()[1] > 0.94 ``` If everything went well, we should see an accuracy value that is around 0.96, which means that we are able to accurately predict the digit in 96% of test images. This is a pretty good result. But as we will see in the next part of this tutorial, we can do a lot better than that. @@ -237,7 +243,7 @@ net = Net() **Figure 3:** First conv + pooling layer in LeNet. -Now we train LeNet with the same hyper-parameters as before. Note that, if a GPU is available, we recommend using it. This greatly speeds up computation given that LeNet is more complex and compute-intensive than the previous multilayer perceptron. To do so, we only need to change `mx.cpu()` to `mx.gpu()` and MXNet takes care of the rest. Just like before, we'll stop training after 10 epochs. +Now we train LeNet with similar hyper-parameters as before. Note that, if a GPU is available, we recommend using it. This greatly speeds up computation given that LeNet is more complex and compute-intensive than the previous multilayer perceptron. To do so, we only need to change `mx.cpu()` to `mx.gpu()` and MXNet takes care of the rest. Just like before, we'll stop training after 10 epochs. Training and prediction can be done in the similar way as we did for MLP. @@ -246,9 +252,11 @@ Training and prediction can be done in the similar way as we did for MLP. We will initialize the network parameters as follows: ```python -ctx = [mx.cpu(0), mx.cpu(1)] + +# set the context on GPU is available otherwise CPU +ctx = [mx.gpu() if mx.test_utils.list_gpus() else mx.cpu()] net.collect_params().initialize(mx.init.Xavier(magnitude=2.24), ctx=ctx) -trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.1}) +trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.03}) ``` #### Training @@ -256,6 +264,7 @@ trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.1}) ```python # Use Accuracy as the evaluation metric. metric = mx.metric.Accuracy() +softmax_cross_entropy_loss = gluon.loss.SoftmaxCrossEntropyLoss() for i in range(epoch): # Reset the train data iterator. @@ -274,9 +283,9 @@ for i in range(epoch): for x, y in zip(data, label): z = net(x) # Computes softmax cross entropy loss. - loss = gluon.loss.softmax_cross_entropy_loss(z, y) + loss = softmax_cross_entropy_loss(z, y) # Backpropogate the error for one iteration. - ag.backward([loss]) + loss.backward() outputs.append(z) # Updates internal evaluation metric.update(label, outputs) diff --git a/docs/tutorials/python/linear-regression.md b/docs/tutorials/python/linear-regression.md index 0a5e308f04f..05afd032ed1 100644 --- a/docs/tutorials/python/linear-regression.md +++ b/docs/tutorials/python/linear-regression.md @@ -22,6 +22,9 @@ To begin, the following code imports the necessary packages we'll need for this import mxnet as mx import numpy as np +# Fix the random seed +mx.random.seed(42) + import logging logging.getLogger().setLevel(logging.DEBUG) ``` @@ -155,7 +158,7 @@ parameters of the model to fit the training data. This is accomplished using the ```python model.fit(train_iter, eval_iter, - optimizer_params={'learning_rate':0.005, 'momentum': 0.9}, + optimizer_params={'learning_rate':0.01, 'momentum': 0.9}, num_epoch=20, eval_metric='mse', batch_end_callback = mx.callback.Speedometer(batch_size, 2)) @@ -175,7 +178,8 @@ evaluating our model's mean squared error (MSE) on the evaluation data. ```python metric = mx.metric.MSE() -model.score(eval_iter, metric) +mse = model.score(eval_iter, metric) +print("Achieved {0:.6f} validation MSE".format(mse[0][1])) assert model.score(eval_iter, metric)[0][1] < 0.01001, "Achieved MSE (%f) is larger than expected (0.01001)" % model.score(eval_iter, metric)[0][1] ``` diff --git a/docs/tutorials/python/matrix_factorization.md b/docs/tutorials/python/matrix_factorization.md index d75da6a2697..154fa4b3e12 100644 --- a/docs/tutorials/python/matrix_factorization.md +++ b/docs/tutorials/python/matrix_factorization.md @@ -5,7 +5,7 @@ that each users have rated some items in the system, we would like to predict how the users would rate the items that they have not yet rated, such that we can make recommendations to the users. -Matrix factorization is one of the mainly used algorithm in recommendation +Matrix factorization is one of the main algorithms used in recommendation systems. It can be used to discover latent features underlying the interactions between two different kinds of entities. @@ -18,6 +18,25 @@ latent features using multi-layer neural networks. In this tutorial, we will work though the steps to implement these ideas in MXNet. +```python +# Set the logging level +import logging +head = '%(asctime)-15s %(message)s' +logging.basicConfig(level=logging.INFO) +``` + +```python +import mxnet as mx +import random + +# Fix the random seeds +mx.random.seed(42) +random.seed(42) + +# set the context on GPU is available otherwise CPU +ctx = mx.gpu() if mx.test_utils.list_gpus() else mx.cpu() +``` + ## Prepare Data We use the [MovieLens](http://grouplens.org/datasets/movielens/) data here, but @@ -28,6 +47,7 @@ provides name and shape information to MXNet about the data and label. ```python + class Batch(object): def __init__(self, data_names, data, label_names, label): self.data = data @@ -48,8 +68,6 @@ Then we define a data iterator, which returns a batch of tuples each time. ```python -import mxnet as mx -import random class Batch(object): def __init__(self, data_names, data, label_names, label): @@ -71,7 +89,7 @@ class DataIter(mx.io.DataIter): super(DataIter, self).__init__() self.batch_size = batch_size self.data = [] - for line in file(fname): + for line in open(fname): tks = line.strip().split('\t') if len(tks) != 4: continue @@ -80,7 +98,7 @@ class DataIter(mx.io.DataIter): self.provide_label = [('score', (self.batch_size, ))] def __iter__(self): - for k in range(len(self.data) / self.batch_size): + for k in range(int(len(self.data) / self.batch_size)): users = [] items = [] scores = [] @@ -110,12 +128,11 @@ Now we download the data and provide a function to obtain the data iterator: import os import urllib import zipfile -if not os.path.exists('ml-100k.zip'): - urllib.urlretrieve('http://files.grouplens.org/datasets/movielens/ml-100k.zip', 'ml-100k.zip') +file = mx.test_utils.download('http://files.grouplens.org/datasets/movielens/ml-100k.zip', 'ml-100k.zip') with zipfile.ZipFile("ml-100k.zip","r") as f: - f.extractall("./") + f.extractall(".") def get_data(batch_size): - return (DataIter('./ml-100k/u1.base', batch_size), DataIter('./ml-100k/u1.test', batch_size)) + return (DataIter(os.path.join('.','ml-100k','u1.base'), batch_size), DataIter(os.path.join('.','ml-100k','u1.test'), batch_size)) ``` Finally we calculate the numbers of users and items for later use. @@ -124,14 +141,14 @@ Finally we calculate the numbers of users and items for later use. def max_id(fname): mu = 0 mi = 0 - for line in file(fname): + for line in open(fname): tks = line.strip().split('\t') if len(tks) != 4: continue mu = max(mu, int(tks[0])) mi = max(mi, int(tks[1])) return mu + 1, mi + 1 -max_user, max_item = max_id('./ml-100k/u.data') +max_user, max_item = max_id(os.path.join('.','ml-100k','u.data')) (max_user, max_item) ``` @@ -157,25 +174,19 @@ classification application. ```python def train(network, batch_size, num_epoch, learning_rate): - model = mx.model.FeedForward( - ctx = mx.gpu(0), - symbol = network, - num_epoch = num_epoch, - learning_rate = learning_rate, - wd = 0.0001, - momentum = 0.9) + model = mx.mod.Module(symbol=network, context=ctx, data_names=('user','item'), label_names=['score']) batch_size = 64 train, test = get_data(batch_size) - import logging - head = '%(asctime)-15s %(message)s' - logging.basicConfig(level=logging.DEBUG) - - model.fit(X = train, + model.fit(train, eval_data = test, eval_metric = RMSE, - batch_end_callback=mx.callback.Speedometer(batch_size, 20000/batch_size),) + batch_end_callback=mx.callback.Speedometer(batch_size, 20000/batch_size), + num_epoch=num_epoch, + optimizer='sgd', + optimizer_params={'learning_rate':learning_rate, 'momentum':0.9, 'wd':0.0001} + ) ``` ## Networks diff --git a/docs/tutorials/python/mnist.md b/docs/tutorials/python/mnist.md index e408ead5aec..df949d487b6 100644 --- a/docs/tutorials/python/mnist.md +++ b/docs/tutorials/python/mnist.md @@ -28,6 +28,12 @@ The following source code downloads and loads the images and the corresponding l ```python import mxnet as mx mnist = mx.test_utils.get_mnist() + +# Fix the seed +mx.random.seed(42) + +# Set the compute context, GPU is available otherwise CPU +ctx = mx.gpu() if mx.test_utils.list_gpus() else mx.cpu() ``` After running the above source code, the entire MNIST dataset should be fully loaded into memory. Note that for large datasets it is not feasible to pre-load the entire dataset first like we did here. What is needed is a mechanism by which we can quickly and efficiently stream data directly from the source. MXNet Data iterators come to the rescue here by providing exactly that. Data iterator is the mechanism by which we feed input data into an MXNet training algorithm and they are very simple to initialize and use and are optimized for speed. During training, we typically process training samples in small batches and over the entire training lifetime will end up processing each training example multiple times. In this tutorial, we'll configure the data iterator to feed examples in batches of 100. Keep in mind that each example is a 28x28 grayscale image and the corresponding label. @@ -97,8 +103,8 @@ Typically, one runs the training until convergence, which means that we have lea ```python import logging logging.getLogger().setLevel(logging.DEBUG) # logging to stdout -# create a trainable module on CPU -mlp_model = mx.mod.Module(symbol=mlp, context=mx.cpu()) +# create a trainable module on compute context +mlp_model = mx.mod.Module(symbol=mlp, context=ctx) mlp_model.fit(train_iter, # train data eval_data=val_iter, # validation data optimizer='sgd', # use SGD to train @@ -164,8 +170,7 @@ lenet = mx.sym.SoftmaxOutput(data=fc2, name='softmax') Now we train LeNet with the same hyper-parameters as before. Note that, if a GPU is available, we recommend using it. This greatly speeds up computation given that LeNet is more complex and compute-intensive than the previous multilayer perceptron. To do so, we only need to change `mx.cpu()` to `mx.gpu()` and MXNet takes care of the rest. Just like before, we'll stop training after 10 epochs. ```python -# create a trainable module on CPU, change to mx.gpu() if GPU is available -lenet_model = mx.mod.Module(symbol=lenet, context=mx.cpu()) +lenet_model = mx.mod.Module(symbol=lenet, context=ctx) # train with the same lenet_model.fit(train_iter, eval_data=val_iter, diff --git a/docs/tutorials/python/predict_image.md b/docs/tutorials/python/predict_image.md index afd2bd71935..c78a5d5a98b 100644 --- a/docs/tutorials/python/predict_image.md +++ b/docs/tutorials/python/predict_image.md @@ -31,12 +31,16 @@ path='http://data.mxnet.io/models/imagenet-11k/' mx.test_utils.download(path+'synset.txt')] ``` -Next, we load the downloaded model. *Note:* If GPU is available, we can replace all -occurrences of `mx.cpu()` with `mx.gpu()` to accelerate the computation. +Next, we load the downloaded model. + +```python +# set the context on CPU, switch to GPU if there is one available +ctx = mx.cpu() +``` ```python sym, arg_params, aux_params = mx.model.load_checkpoint('resnet-152', 0) -mod = mx.mod.Module(symbol=sym, context=mx.cpu(), label_names=None) +mod = mx.mod.Module(symbol=sym, context=ctx, label_names=None) mod.bind(for_training=False, data_shapes=[('data', (1,3,224,224))], label_shapes=mod._label_shapes) mod.set_params(arg_params, aux_params, allow_missing=True) @@ -89,11 +93,11 @@ def predict(url): Now, we can perform prediction with any downloadable URL: ```python -predict('http://writm.com/wp-content/uploads/2016/08/Cat-hd-wallpapers.jpg') +predict('https://github.com/dmlc/web-data/blob/master/mxnet/doc/tutorials/python/predict_image/cat.jpg?raw=true') ``` ```python -predict('http://thenotoriouspug.com/wp-content/uploads/2015/01/Pug-Cookie-1920x1080-1024x576.jpg') +predict('https://github.com/dmlc/web-data/blob/master/mxnet/doc/tutorials/python/predict_image/dog.jpg?raw=true') ``` ## Feature extraction @@ -120,7 +124,7 @@ outputs the flattened layer and creates a model. ```python fe_sym = all_layers['flatten0_output'] -fe_mod = mx.mod.Module(symbol=fe_sym, context=mx.cpu(), label_names=None) +fe_mod = mx.mod.Module(symbol=fe_sym, context=ctx, label_names=None) fe_mod.bind(for_training=False, data_shapes=[('data', (1,3,224,224))]) fe_mod.set_params(arg_params, aux_params) ``` @@ -128,7 +132,7 @@ fe_mod.set_params(arg_params, aux_params) We can now invoke `forward` to obtain the features: ```python -img = get_image('http://writm.com/wp-content/uploads/2016/08/Cat-hd-wallpapers.jpg') +img = get_image('https://github.com/dmlc/web-data/blob/master/mxnet/doc/tutorials/python/predict_image/cat.jpg?raw=true') fe_mod.forward(Batch([mx.nd.array(img)])) features = fe_mod.get_outputs()[0].asnumpy() print(features) diff --git a/docs/tutorials/sparse/row_sparse.md b/docs/tutorials/sparse/row_sparse.md index ffc683f29a7..dc6fee7dc58 100644 --- a/docs/tutorials/sparse/row_sparse.md +++ b/docs/tutorials/sparse/row_sparse.md @@ -174,9 +174,7 @@ b = mx.nd.sparse.row_sparse_array((data_np, indices_np), shape=shape) - {'a': - <RowSparseNDArray 6x2 @cpu(0)>, 'b': - <RowSparseNDArray 6x2 @cpu(0)>} +`{'a': <RowSparseNDArray 6x2 @cpu(0)>, 'b': <RowSparseNDArray 6x2 @cpu(0)>}`<!--notebook-skip-line--> @@ -210,7 +208,7 @@ d = mx.nd.array(a, dtype=np.float16) - (numpy.float32, numpy.float16) +`(numpy.float32, numpy.float16)`<!--notebook-skip-line--> @@ -351,7 +349,7 @@ g.copyto(f) - {'e.stype': 'row_sparse', 'f.stype': 'row_sparse', 'g.stype': 'default'} +`{'e.stype': 'row_sparse', 'f.stype': 'row_sparse', 'g.stype': 'default'}`<!--notebook-skip-line--> @@ -429,7 +427,7 @@ c = a + mx.nd.ones((5, 2)) # c will be a dense NDArray - {'b.stype': 'row_sparse', 'c.stype': 'default'} +`{'b.stype': 'row_sparse', 'c.stype': 'default'}`<!--notebook-skip-line--> @@ -453,7 +451,7 @@ e = mx.nd.log(a, out=e) # dense operator with a sparse output - {'a.stype': 'row_sparse', 'd.stype': 'default', 'e.stype': 'row_sparse'} +`{'a.stype': 'row_sparse', 'd.stype': 'default', 'e.stype': 'row_sparse'}` <!--notebook-skip-line--> diff --git a/docs/tutorials/sparse/train.md b/docs/tutorials/sparse/train.md index e31f0465da9..6475f2e8907 100644 --- a/docs/tutorials/sparse/train.md +++ b/docs/tutorials/sparse/train.md @@ -186,14 +186,14 @@ fallback_log = fallback_exec.outputs[1] When the environment variable `MXNET_INFER_STORAGE_TYPE_VERBOSE_LOGGING` is set to `1`, MXNet will log the storage type information of operators' inputs and outputs in the computation graph. For example, we can inspect the storage types of -a linear classification network with sparse operators as follows: +a linear classification network with sparse operators. Uncomment the line below and inspect your console.: ```python # Set logging level for executor import mxnet as mx import os -os.environ['MXNET_INFER_STORAGE_TYPE_VERBOSE_LOGGING'] = "1" +#os.environ['MXNET_INFER_STORAGE_TYPE_VERBOSE_LOGGING'] = "1" # Data in csr format data = mx.sym.var('data', stype='csr', shape=(32, 10000)) # Weight in row_sparse format @@ -299,16 +299,7 @@ for epoch in range(10): assert metric.get()[1] < 1, "Achieved MSE (%f) is larger than expected (1.0)" % metric.get()[1] ``` - Epoch 0, Metric = ('mse', 886.16457029229127) - Epoch 1, Metric = ('mse', 173.16523056503445) - Epoch 2, Metric = ('mse', 71.625164168341811) - Epoch 3, Metric = ('mse', 29.625375983519298) - Epoch 4, Metric = ('mse', 12.45004676561909) - Epoch 5, Metric = ('mse', 6.9090727975622368) - Epoch 6, Metric = ('mse', 3.0759215722750142) - Epoch 7, Metric = ('mse', 1.3106610134811276) - Epoch 8, Metric = ('mse', 0.63063102482907718) - Epoch 9, Metric = ('mse', 0.35979430613957991) +`Epoch 9, Metric = ('mse', 0.35979430613957991)`<!--notebook-skip-line--> diff --git a/tests/nightly/test_tutorial.py b/tests/nightly/test_tutorial.py index ff7ee9db678..3c9470c7104 100644 --- a/tests/nightly/test_tutorial.py +++ b/tests/nightly/test_tutorial.py @@ -25,87 +25,114 @@ import os import warnings import imp - +import shutil +import time +import argparse import traceback import nbformat from nbconvert.preprocessors import ExecutePreprocessor +import sys fail_dict = {} +TIME_OUT = 1800 -def test_tutorial(file_path): - """Run tutorial python script and save any error or warning. - If no error or warning occurs, run notebook. - - Parameters - ---------- - file_path : str - path of tutorial markdown file - """ - with warnings.catch_warnings(record=True) as w: - tutorial_name = os.path.basename(file_path) - print file_path + '.py' - try: - imp.load_source('tutorial', file_path + '.py') - if len(w) > 0: - err_msg = "%s.py has %d warnings.\n" % (tutorial_name, len(w)) - fail_dict[tutorial_name] = err_msg - else: - test_tutorial_nb(file_path) - except Exception: - err_msg = "%s.py has error:\n%s" % (tutorial_name, traceback.format_exc()) - fail_dict[tutorial_name] = err_msg - -def test_tutorial_nb(file_path): +def test_tutorial_nb(file_path, workingdir, kernel=None): """Run tutorial jupyter notebook to catch any execution error. Parameters ---------- file_path : str - path of tutorial markdown file + path of tutorial .ipynb file + workingdir: str + path of the directory to run the tutorial in + kernel: str + Default None + name of the kernel to use, if none, will use first kernel + in the list """ tutorial_name = os.path.basename(file_path) + sys.stdout.write('Testing {}...'.format(file_path)) + sys.stdout.flush() + tick = time.time() notebook = nbformat.read(file_path + '.ipynb', as_version=4) - eprocessor = ExecutePreprocessor(timeout=1800) + if kernel: + eprocessor = ExecutePreprocessor(timeout=TIME_OUT, kernel_name=kernel) + else: + eprocessor = ExecutePreprocessor(timeout=TIME_OUT) + success = True try: - eprocessor.preprocess(notebook, {'metadata': {}}) + eprocessor.preprocess(notebook, {'metadata': {'path':workingdir}}) except Exception as err: err_msg = str(err) fail_dict[tutorial_name] = err_msg + success = False finally: - output_nb = open("output.txt", mode='w') + output_file = os.path.join(workingdir, "output.txt") + output_nb = open(output_file, mode='w') nbformat.write(notebook, output_nb) output_nb.close() - output_nb = open("output.txt", mode='r') + output_nb = open(output_file, mode='r') for line in output_nb: if "Warning:" in line: - fail_dict[tutorial_name] = "%s has warning." % (tutorial_name) - return + success = False + if tutorial_name in fail_dict: + fail_dict[tutorial_name] += "\n"+line + else: + fail_dict[tutorial_name] = "Warning:\n"+line + sys.stdout.write(' Elapsed time: {0:.2f}s '.format(time.time()-tick )) + sys.stdout.write(' [{}] \n'.format('Success' if success else 'Failed')) + sys.stdout.flush() if __name__ == "__main__": - tutorial_dir = '../../docs/_build/html/tutorials/' - with open('test_tutorial_config.txt') as config_file: - tutorial_list = [] - for line in config_file: - tutorial_list.append(line.lstrip().rstrip()) - file_dir = tutorial_dir + line.lstrip().rstrip() - test_tutorial_nb(file_dir) + tutorial_dir = os.path.join('..','..','docs', '_build', 'html', 'tutorials') + tick = time.time() + + parser = argparse.ArgumentParser() + parser.add_argument("--tutorial", help="tutorial to test, if not set, read from test_tutorial_config.txt") + parser.add_argument("--kernel", help="name of the jupyter kernel to use for the test") + parser.add_argument("--no-cache", help="clean the temp directory", action="store_true", dest="no_cache") + args = parser.parse_args() + + + tutorial_list = [] + if args.tutorial: + tutorial_list.append(args.tutorial) + else: + with open('test_tutorial_config.txt') as config_file: + for line in config_file: + tutorial_list.append(line.lstrip().rstrip()) + + temp_dir = 'tmp_notebook' + if args.no_cache: + print("Cleaning and setting up temp directory '{}'".format(temp_dir)) + shutil.rmtree(temp_dir, ignore_errors=True) + + kernel = args.kernel if args.kernel else None + + for tutorial in tutorial_list: + file_dir = os.path.join(*([tutorial_dir]+tutorial.split('/'))) + working_dir = os.path.join(*([temp_dir]+tutorial.split('/'))) + if not os.path.isdir(working_dir): + os.makedirs(working_dir) + test_tutorial_nb(file_dir, working_dir, kernel) - fail_num = len(fail_dict) - success_num = len(tutorial_list) - fail_num - print "Test Summary Start" - print "%d tutorials tested:" % (len(tutorial_list)) - for tutorial in tutorial_list: - print tutorial - print "\n%d tests failed:" % (fail_num) - for tutorial, msg in fail_dict.items(): - print tutorial + ":" - print msg - print "Test Summary End" - print "Stats start" - print "[Passed: %d of %d]" % (success_num, len(tutorial_list)) - print "Stats end" + fail_num = len(fail_dict) + success_num = len(tutorial_list) - fail_num + print("Test Summary Start") + print("%d tutorials tested:" % (len(tutorial_list))) + for tutorial in tutorial_list: + print(tutorial) + print("\n%d tests failed:" % (fail_num)) + for tutorial, msg in fail_dict.items(): + print(tutorial + ":") + print(msg) + print("Test Summary End") + print("Stats start") + print("[Passed: %d of %d]" % (success_num, len(tutorial_list))) + print("Total time: {:.2f}s".format(time.time()-tick)) + print("Stats end") - if fail_num > 0: - exit(1) + if fail_num > 0: + exit(1) diff --git a/tests/nightly/test_tutorial_config.txt b/tests/nightly/test_tutorial_config.txt index 428309b84c8..e88074f99f3 100644 --- a/tests/nightly/test_tutorial_config.txt +++ b/tests/nightly/test_tutorial_config.txt @@ -5,3 +5,16 @@ basic/data python/linear-regression python/mnist python/predict_image +onnx/super_resolution +onnx/fine_tuning_gluon +onnx/inference_on_onnx_model +basic/ndarray_indexing +python/matrix_factorization +gluon/ndarray +gluon/mnist +gluon/autograd +gluon/gluon +gluon/hybrid +sparse/row_sparse +sparse/csr +sparse/train \ No newline at end of file ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
