Ishitori commented on a change in pull request #10607: New tutorial on how to create a new custom layer in Gluon URL: https://github.com/apache/incubator-mxnet/pull/10607#discussion_r183537061
########## File path: docs/tutorials/python/custom_layer.md ########## @@ -0,0 +1,247 @@ + +# How to write a custom layer in Apache MxNet Gluon API + +While Gluon API for Apache MxNet comes with [a decent number of predefined layers](https://mxnet.incubator.apache.org/api/python/gluon/nn.html), at some point one may find that a new layer is needed. Adding a new layer in Gluon API is straightforward, yet there are a few things that one needs to keep in mind. + +In this article, I will cover how to create a new layer from scratch, how to use it, what are possible pitfalls and how to avoid them. + +## The simplest custom layer + +To create a new layer in Gluon API, one must create a class that inherits from [Block](https://mxnet.incubator.apache.org/api/python/gluon/gluon.html#mxnet.gluon.Block) class. This class provides the most basic functionality, and all predefined layers inherit from it directly or via other subclasses. Because each layer in Apache MxNet inherits from `Block`, words "layer" and "block" are used interchangeably inside of the Apache MxNet community. + +The only instance method needed to be implemented is [forward()](https://mxnet.incubator.apache.org/api/python/gluon/gluon.html#mxnet.gluon.Block.forward), which defines what exactly your layer is going to do during forward propagation. Notice, that it doesn't require to provide what the block should do during backpropagation. Backpropagation pass for blocks is done by Apache MxNet for you. + +In the example below, we define a new layer and implement `forward()` method to normalize input data by fitting it into a range of [0, 1]. + + +```python +# Do some initial imports used throughout this tutorial +from __future__ import print_function +import mxnet as mx +from mxnet import nd, gluon, autograd +from mxnet.gluon.nn import Dense +mx.random.seed(1) # Set seed for reproducable results +``` + + +```python +class NormalizationLayer(gluon.Block): + def __init__(self): + super(NormalizationLayer, self).__init__() + + def forward(self, x): + return (x - nd.min(x)) / (nd.max(x) - nd.min(x)) +``` + +The rest of methods of the `Block` class are already implemented, and majority of them are used to work with parameters of a block. There is one very special method named [hybridize()](https://mxnet.incubator.apache.org/api/python/gluon/gluon.html#mxnet.gluon.Block.hybridize), though, which I am going to cover before moving to a more complex example of a custom layer. + +## Hybridization and the difference between Block and HybridBlock + +Looking into the implementation of [existing layers](https://mxnet.incubator.apache.org/api/python/gluon/nn.html), one may find that more often a block inherits from a [HybridBlock](https://mxnet.incubator.apache.org/api/python/gluon/gluon.html#mxnet.gluon.HybridBlock), instead of directly inheriting from `Block` class. + +The reason for that is that `HybridBlock` allows to write custom layers that can be used in imperative programming as well as in symbolic programming. It is convinient to support both ways, because of the different values these programming models bring. The imperative programming eases the debugging of the code - one can use regular debugging tools available in modern IDEs to go line by line through the computation. The symbolic programming provides faster execution speed, but harder to debug. You can learn more about the difference between symbolic vs. imperative programming from [this article](https://mxnet.incubator.apache.org/architecture/program_model.html). + +Because of these reasons it is recommended to develop a new layer using imperative model, but deploy it using symbolic model. + +Hybridization is a process that Apache MxNet uses to create a symbolic graph of a forward computation. Optimization of this computational graph allows to increase performance. Once the symbolic graph is created, Apache MxNet caches and reuses it for subsequent computations. + +To simplify support of both imperative and symbolic programming, Apache MxNet introduce the `HybridBlock` class. Compare to the `Block` class, `HybridBlock` already has its [forward()](https://mxnet.incubator.apache.org/api/python/gluon/gluon.html#mxnet.gluon.HybridBlock.forward) method implemented, but it defines a [hybrid_forward()](https://mxnet.incubator.apache.org/api/python/gluon/gluon.html#mxnet.gluon.HybridBlock.hybrid_forward) method that needs to be implemented. + +From API point of view, the main difference between `forward()` and `hybrid_forward()` is an `F` argument. This argument sometimes is refered as a `backend` in the Apache MxNet community. Depending on if hybridization has been done or not, `F` can refer either to [mxnet.ndarray API](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html) or [mxnet.symbol API](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html). The former is used for imperative programming, and the latter for symbolic programming. + +To support hybridization, it is important to use only methods avaible directly from `F`. Usually, there are equivalent methods in both APIs, but sometimes there are mismatches or small variations. For example, by default, subtraction and division of NDArrays support broadcasting, while in Symbol API broadcasting is supported in separate operators. + +Knowing this, we can can rewrite our example layer, using HybridBlock: + + +```python +class NormalizationHybridLayer(gluon.HybridBlock): + def __init__(self): + super(NormalizationHybridLayer, self).__init__() + + def hybrid_forward(self, F, x): + return F.broadcast_div(F.broadcast_sub(x, F.min(x)), (F.broadcast_sub(F.max(x), F.min(x)))) +``` + +Thanks to inheriting from HybridBlock, one can easily do forward pass on a given ndarray, either on CPU or GPU. Notice that we don't call `forward()` or `hybrid_forward()` methods directly. + + +```python +layer = NormalizationHybridLayer() +layer(nd.array([1, 2, 3], ctx=mx.cpu())) +``` + + + + + + [0. 0.5 1. ] + <NDArray 3 @cpu(0)> + + + +As a rule of thumb, one should always implement custom layers by inheriting from `HybridBlock`. This eaeses the development, and doesn't affect execution speed once hybridization is done. Review comment: Reformulate it as: This eases the debugging, though require to think of different backends. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
