piiswrong closed pull request #9869: Exception handling documentation URL: https://github.com/apache/incubator-mxnet/pull/9869
This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/dmlc-core b/dmlc-core index a1fd6834c0c..282b98663f5 160000 --- a/dmlc-core +++ b/dmlc-core @@ -1 +1 @@ -Subproject commit a1fd6834c0cd3fd2cc586deec2dc24194924cada +Subproject commit 282b98663f59df6b26f906580af610dea3046f22 diff --git a/docs/architecture/exception_handling.md b/docs/architecture/exception_handling.md new file mode 100644 index 00000000000..5b4448a53ab --- /dev/null +++ b/docs/architecture/exception_handling.md @@ -0,0 +1,111 @@ +# Exception Handling in MXNet + +This tutorial explains the exception handling support in MXNet, +and provides examples on how to throw and handle exceptions when in a multithreaded context. +Although, the examples are in Python, they can be easily extended to MXNet +language bindings. + +MXNet exceptions can be thrown from two areas: +- MXNet main thread. For eg. Infershape and InferType. +- Spawned threads: + * By dependency engine for operator execution in parallel + * By the iterators, during the data loading, text parsing phase etc. + +In the first case, the exception is thrown and can be handled in the main thread. +In the second case, the exception is thrown in a spawned thread, caught and transported to the +main thread, where it is rethrown. This tutorial will give more explanation and examples on how +to handle exceptions for the second case. + +## Prerequisites + +To complete this tutorial, we need: +- MXNet [7b24137](https://github.com/apache/incubator-mxnet/commit/7b24137ed45df605defa4ce72ec91554f6e445f0). See Instructions in [Setup and Installation](http://mxnet.io/install/index.html). + +## Exception Handling for Iterators + +The below example shows how to handle exceptions for iterators. In this example, +we populate files for data and labels with fewer number of labels compared to the +number of samples. This should throw an exception. + +CSVIter uses PrefetcherIter for loading and parsing data. +The PrefetcherIter spawns a producer thread in the background which prefetches +the data while the main thread consumes the data. The exception is thrown in the spawned +producer thread during the prefetching, when the label is not found corresponding to a specific sample. + +The exception is transported to the main thread, where it is rethrown when Next is +called as part of the following line: `for batch in iter(data_train)`. + +In general, Exception may be rethrown as part of `Next` and `BeforeFirst` calls which correspond to `reset()` and `next()` methods in `MXDataIter` for Python language bindings. + +```python +import os +import mxnet as mx + +cwd = os.getcwd() +data_path = os.path.join(cwd, "data.csv") +label_path = os.path.join(cwd, "label.csv") + +with open(data_path, "w") as fout: + for i in range(8): + fout.write("1,2,3,4,5,6,7,8,9,10\n") + +with open(label_path, "w") as fout: + for i in range(7): + fout.write("label"+str(i)) + +try: + data_train = mx.io.CSVIter(data_csv=data_path, label_csv=label_path, data_shape=(1, 10), + batch_size=4) + + for batch in iter(data_train): + print(data_train.getdata().asnumpy()) +except mx.base.MXNetError as ex: + print("Exception handled") + print(ex) +``` + +### Limitation + +There is a race condition when your last `next()` call doesnt reach the batch in your dataset where exception occurs. Exception may or may not be thrown in this case depending on which thread wins the race. To avoid this situation, you should try and iterate through your full dataset if you think it can throw exceptions which need to be handled. + + +## Exception Handling for Operators + +The below example shows how to handle exceptions for operators in the imperative mode. + +For the operator case, the dependency engine spawns a number of threads if it is running in the `ThreadedEnginePool` or `ThreadedEnginePerDevice` mode. The final operator is executed in one of the spawned threads. + +If an operator throws an exception during execution, this exception is propagated +down the dependency chain. Once there is a synchronizing call i.e. WaitToRead for a variable in the dependency chain, the propagated exception is rethrown. + +In the below example, I illustrate how an exception that occured in the first line is propagated down the dependency chain, and finally is rethrown when we make a synchronizing call to WaitToRead. + +```python +import mxnet as mx +a = mx.nd.random.normal(0, 1, (2, 2)) +b = mx.nd.random.normal(0, 2, (2, 2)) +c = mx.nd.dot(a, b) +d = mx.nd.random.normal(0, -1, (2, 2)) +e = mx.nd.dot(c, d) +e.wait_to_read() +``` + +Although the above exception occurs when executing the operation which writes to the variable d in one of the child threads, it is thrown only when the synchronization happens as part of the line: `e.wait_to_read()`. + +Let us take another example. In the following case, we write to two variables and then `wait_to_read` for both. This example shows that any particular exception will not be thrown more than once. + +```python +import mxnet as mx +a = mx.nd.random.normal(0, 1, (2, 2)) +b = mx.nd.random.normal(0, -1, (2, 2)) +c, d = mx.nd.dot(a, b) +try: + c.asnumpy() +except mx.base.MXNetError as ex: + print("Exception handled") +d.asnumpy() +``` + +### Limitation + +Rethrowing exceptions as part of `mx.nd.waitall` is not supported. So if your code executes a few operators and then calls `waitall` instead of `wait_to_read`/`asnumpy`, the exception will disappear. Please avoid waitalls in your code unless you are confident about your code not throwing exception in any scenario. diff --git a/docs/architecture/index.md b/docs/architecture/index.md index 7a8ec3db048..91fb5f51d7b 100644 --- a/docs/architecture/index.md +++ b/docs/architecture/index.md @@ -20,3 +20,4 @@ Additionally, we provide an overview of the complete MXNet system. * [Dependency Engine for Deep Learning](http://mxnet.io/architecture/note_engine.html) * [Optimizing the Memory Consumption in Deep Learning](http://mxnet.io/architecture/note_memory.html) * [Efficient Data Loading Module for Deep Learning](http://mxnet.io/architecture/note_data_loading.html) +* [Exception Handling in MXNet](http://mxnet.io/architecture/exception_handling.html) diff --git a/tests/python/unittest/test_io.py b/tests/python/unittest/test_io.py index 58ca1d74fbb..4e23a22e09e 100644 --- a/tests/python/unittest/test_io.py +++ b/tests/python/unittest/test_io.py @@ -18,6 +18,7 @@ # pylint: skip-file import mxnet as mx from mxnet.test_utils import * +from mxnet.base import MXNetError import numpy as np import os, gzip import pickle as pickle @@ -249,8 +250,31 @@ def check_libSVMIter_news_data(): assert(num_batches == int(expected_num_batches)), num_batches data_train.reset() + def check_libSVMIter_exception(): + cwd = os.getcwd() + data_path = os.path.join(cwd, 'data.t') + label_path = os.path.join(cwd, 'label.t') + with open(data_path, 'w') as fout: + fout.write('1.0 0:0.5 2:1.2\n') + fout.write('-2.0\n') + # Below line has a neg indice. Should throw an exception + fout.write('-3.0 -1:0.6 1:2.4 2:1.2\n') + fout.write('4 2:-1.2\n') + + with open(label_path, 'w') as fout: + fout.write('1.0\n') + fout.write('-2.0 0:0.125\n') + fout.write('-3.0 2:1.2\n') + fout.write('4 1:1.0 2:-1.2\n') + data_dir = os.path.join(cwd, 'data') + data_train = mx.io.LibSVMIter(data_libsvm=data_path, label_libsvm=label_path, + data_shape=(3, ), label_shape=(3, ), batch_size=3) + for batch in iter(data_train): + data_train.get_data().asnumpy() + check_libSVMIter_synthetic() check_libSVMIter_news_data() + assertRaises(MXNetError, check_libSVMIter_exception) def test_DataBatch(): ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services