thomelane commented on a change in pull request #15427: [TUTORIAL] Gluon performance tips and tricks URL: https://github.com/apache/incubator-mxnet/pull/15427#discussion_r299723669
########## File path: docs/tutorials/gluon/performance.md ########## @@ -0,0 +1,483 @@ +<!--- Licensed to the Apache Software Foundation (ASF) under one --> +<!--- or more contributor license agreements. See the NOTICE file --> +<!--- distributed with this work for additional information --> +<!--- regarding copyright ownership. The ASF licenses this file --> +<!--- to you under the Apache License, Version 2.0 (the --> +<!--- "License"); you may not use this file except in compliance --> +<!--- with the License. You may obtain a copy of the License at --> + +<!--- http://www.apache.org/licenses/LICENSE-2.0 --> + +<!--- Unless required by applicable law or agreed to in writing, --> +<!--- software distributed under the License is distributed on an --> +<!--- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY --> +<!--- KIND, either express or implied. See the License for the --> +<!--- specific language governing permissions and limitations --> +<!--- under the License. --> + +# Gluon Performance Tips & Tricks + +Compared to traditional machine learning methods, the field of deep-learning has increased model accuracy across a wide range of tasks, but it has also increased the amount of computation required for model training and inference. Specialised hardware chips, such as GPUs and FPGAs, can speed up the execution of networks, but it can sometimes be hard to write code that uses the hardware to its full potential. We will be looking at a few simple tips and trick in this tutorial that you can use to speed up training and ultimately save on training costs. + +We'll start by writing some code to train an image classification network for the CIFAR-10 dataset, and then benchmark the throughput of the network in terms of samples processed per second. After some performance analysis, we'll identify the bottlenecks (i.e. the components limiting throughput) and improve the training speed step-by-step. We'll bring together all the tips and tricks at the end and evaluate our performance gains. + + +```python +from __future__ import print_function +import multiprocessing +import time +import mxnet as mx +import numpy as np +``` + +An Amazon EC2 p3.2xlarge instance was used to benchmark the code in this tutorial. You are likely to get difference results and find different bottlenecks on other hardware, but these tips and tricks should still help improve training speed for bottleneck components. A GPU is recommended for this example. + + +```python +ctx = mx.gpu() if mx.test_utils.list_gpus() else mx.cpu() +print("Using {} context.".format(ctx)) +``` + + Using gpu(0) context. + + +We'll use the `CIFAR10` dataset provided out-of-the-box with Gluon. + + +```python +dataset = mx.gluon.data.vision.CIFAR10(train=True) +print('{} samples'.format(len(dataset))) +``` + + 50000 samples + + +So we can learn how to identify training bottlenecks, let's intentionally introduce a short `sleep` into the data loading pipeline. We transform each 32x32 CIFAR-10 image to 244x244 so we can use it with the ResNet-50 network designed for ImageNet. [CIFAR-10 specific ResNet networks](https://gluon-cv.mxnet.io/api/model_zoo.html#gluoncv.model_zoo.get_cifar_resnet) exist but we use the more standard ImageNet variants in this example. + + +```python +def transform_fn(x): + time.sleep(0.01) # artificial slow-down + image = mx.image.imresize(x, w=244, h=244) + return image.astype('float32').transpose((2, 0, 1)) + +dataset = dataset.transform_first(transform_fn) +``` + +Setting our batch size to 16, we can create the `DataLoader`. + + +```python +batch_size = 16 +dataloader = mx.gluon.data.DataLoader(dataset, + batch_size=batch_size, + shuffle=True, + last_batch="discard") +print('{} batches'.format(len(dataloader))) +``` + + 3125 batches + + +Up next, we create all of the other components required for training, such as the network, the loss function, the evaluation metric and parameter trainer. + + +```python +net = mx.gluon.model_zoo.vision.resnet50_v2(pretrained=False, ctx=ctx) +net.initialize(mx.init.Xavier(magnitude=2.3), ctx=ctx) +loss_fn = mx.gluon.loss.SoftmaxCrossEntropyLoss() +metric = mx.metric.Accuracy() +learning_rate = 0.001 +trainer = mx.gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': learning_rate}) +``` + +## Initial Benchmark + +As a starting point, let's benchmark the throughput of our training loop: calculating the average samples per second across 25 iterations, where each iteration is a batch of 16 samples. We'll run a single forward pass through the network before starting our benchmark timer to avoid including shape inference and lazy initialization in the throughput calculations. + + +```python +def single_forward(net, dataloader, dtype='float32'): + data, label = next(iter(dataloader)) + data = data.astype(dtype) + data = data.as_in_context(ctx) + pred = net(data) + pred.wait_to_read() +``` + + +```python +single_forward(net, dataloader) +iters = 25 +num_samples = 0 +num_iters = 0 +start_time = time.time() +for iter_idx, (data, label) in enumerate(dataloader): + num_samples += data.shape[0] + num_iters += 1 + data = data.as_in_context(ctx) + label = label.as_in_context(ctx) + with mx.autograd.record(): + pred = net(data) + loss = loss_fn(pred, label) + loss.backward() + trainer.step(data.shape[0]) + metric.update(label, pred) + print('.', end='') + if num_iters >= iters: + break +mx.nd.waitall() +end_time = time.time() +total_time = end_time - start_time +print('\n') +print('average iterations/sec: {:.4f}'.format(num_iters/total_time)) +print('average samples/sec: {:.4f}'.format(num_samples/total_time)) +``` + + ......................... + + average iterations/sec: 4.2862 + average samples/sec: 68.5795 + + +Although ~70 samples per second might sound respectable, let's see if we can do any better by identifying the bottleneck in the training loop and optimizing that component. A significant amount of time can be wasted by optimizing components that aren't bottlenecks. + +## Identifying the bottleneck + +Monitoring the CPU (with `top`) and GPU utilization (with `nvidia-smi`) provide clues as to where potential bottlenecks lie. With the example above, when simultaneously running these monitoring tool, you might spot a single process on the CPU fixed at ~100% utilization while the GPU utilization behaves erratically and often falls to ~0%. Seeing behaviour like can indicate the CPU is struggling to process data and the GPU is being starved of data. + +MXNet's Profiler is another highly recommended tool for identifying bottlenecks, since it gives timing data for individual MXNet operations. Check out this comprehensive tutorial for more details. As a simpler form of analysis, we will split our training loop into two common components: + +1. Data Loading +2. Network Execution (forward and backward passes) + +We define two function to independently benchmark these components: `benchmark_dataloader` and `benchmark_network`. + + +```python +def benchmark_dataloader(dataloader, iters=25): + num_samples = 0 + num_iters = 0 + start_time = time.time() + startup_time = None + for iter_idx, sample in enumerate(dataloader): + if startup_time is None: + startup_time = time.time() + num_samples += sample[0].shape[0] + num_iters += 1 + if num_iters >= iters: + break + print('.', end='') + end_time = time.time() + total_time = end_time - start_time + total_startup_time = startup_time - start_time + total_iter_time = end_time - startup_time + print('\n') + print('total startup time: {:.4f}'.format(total_startup_time)) + print('average iterations/sec: {:.4f}'.format(num_iters/total_iter_time)) + print('average samples/sec: {:.4f}'.format(num_samples/total_iter_time)) + + +def benchmark_network(data, label, net, loss_fn, trainer, iters=25): + num_samples = 0 + num_iters = 0 + mx.nd.waitall() + start_time = time.time() + for iter_idx in range(iters): + num_samples += data.shape[0] + num_iters += 1 + with mx.autograd.record(): + pred = net(data) + loss = loss_fn(pred, label) + loss.backward() + trainer.step(data.shape[0]) + mx.nd.waitall() + if num_iters >= iters: + break + print('.', end='') + end_time = time.time() + total_time = end_time - start_time + print('\n') + print('average iterations/sec: {:.4f}'.format(num_iters/total_time)) + print('average samples/sec: {:.4f}'.format(num_samples/total_time)) +``` + +Our `benchmark_dataloader` function just loops through the `DataLoader` for a given number of iterations: it doesn't transfer the data to the correct context or pass it to the network. Our `benchmark_network` function just performs a forward and backward pass on an identical (and pre-transferred) batch of data: it doesn't require new data to be loaded. We'll run both of these functions now. + + +```python +print('\n', '### benchmark_dataloader', '\n') +benchmark_dataloader(dataloader) +print('\n', '### benchmark_network', '\n') +data, label = next(iter(dataloader)) +data = data.as_in_context(ctx) +label = label.as_in_context(ctx) +benchmark_network(data, label, net, loss_fn, trainer) +``` + + + ### benchmark_dataloader + + ........................ + + total startup time: 0.1723 + average iterations/sec: 6.1231 + average samples/sec: 97.9701 + + ### benchmark_network + + ........................ + + average iterations/sec: 13.6279 + average samples/sec: 218.0460 + + +Our data loading pipeline appears to be the bottleneck for training: ~100 samples/second compared with ~200 samples/second for network execution. One limiting factor could be disk throughput when reading samples (using a SSD instead of HDD can help with this), but in this case we intentionally added a delay in data transformation. Augmentation can often be a bottleneck in training if the following trick isn't applied. + +## Tips & Tricks #1: Use multiple workers on `DataLoader` + +In the previous section, we established that the data loading component of the training loop was the bottleneck. Instead of simply removing the artificial delay, let's assume it was some pre-processing or augmentation step that couldn't be removed. We found that the CPU utilization was fixed at 100%, but this was just for a single core. Usually machines have multiple cores and with one easy trick we can leverage more CPU cores to pre-process the data. Setting `num_workers` on the `DataLoader` will result in multiple workers being used to preprocess the data. We can use `multiprocessing.cpu_count()` to find the number of CPU cores available on the machine, and we save 1 core for the main thread. + + +```python +num_workers = multiprocessing.cpu_count() - 1 +dataloader = mx.gluon.data.DataLoader(dataset, + batch_size=batch_size, + shuffle=True, + last_batch="discard", + num_workers=num_workers) +print('Using {} workers for DataLoader.'.format(num_workers)) +``` + + Using 7 workers for DataLoader. + + +We benchmark the two main components once again: + + +```python +print('\n', '### benchmark_dataloader', '\n') +benchmark_dataloader(dataloader) +print('\n', '### benchmark_network', '\n') +data, label = next(iter(dataloader)) +data = data.as_in_context(ctx) +label = label.as_in_context(ctx) +benchmark_network(data, label, net, loss_fn, trainer, iters=10) +``` + + + ### benchmark_dataloader + + ........................ + + total startup time: 0.1967 + average iterations/sec: 45.6467 + average samples/sec: 730.3466 + + ### benchmark_network + + ......... + + average iterations/sec: 13.2545 + average samples/sec: 212.0723 + + +Our data loading pipeline is no longer the bottleneck for training throughput: ~700 samples per second versus ~200 samples per second for network execution as before. We can now focus our attention on improving the network throughput. Review comment: Added. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
