To better answer Hagay's question, I would like to dive down a bit deeper on the relation between MXNet, NNVM and model exchange format like ONNX.
There are two major trends in deep learning systems now: - Common serializable formats, like ONNX and CoreML, that defines the model exchange format. - The in-memory graph IR for quick optimization and JIT. NNVM, Tensorflow's XLA falls into this category. The exchange formats are great, it only poses a layer of conversion, which is good for exchange. The real meat still comes from the compilation and JIT pipeline you have to offer. For that, we will need an in-memory IR, because of the cost of constructing, serialize could be high for the exchange formats like protobuf. And usually, the exchange formats are designed in a minimalistic fashion, making it less easy to extend more information to support in-depth optimization like automatic quantization, accelerator support. The current MXNet relies on NNVM for in-memory IR manipulation but does not contain a compilation component that compiles to the hardware backends. Doing export to an exchange format and then back into NNVM run the compilation poses too much burden that JIT compiler could pay. Using the same in-memory graph IR as the compilation stack give much more advantage in terms of this. The newly introduces nnvm/top and compiler offers in-memory graph optimization and compilation and offers more hardware backend directly via TVM. We already see promising results in edge deployments with a much lower overhead of runtime. We will further benefit quickly from more graph optimizations that it has to offer. Building support around this new paradigm offers us advantage of being future compatible and takes full benefit of the points I mentioned above Tianqi On Wed, Oct 18, 2017 at 4:57 PM, Lupesko, Hagay <[email protected]> wrote: > Roshani – this is an exciting initiative, ONNX support on MXNet will > enable more users to ramp up on MXNet, which is great. > > Tianqi – a few questions and thoughts about your note: > - “More hardware backends to mxnet” – MXNet users get the same benefit of > HW support implementing ONNX import on top of MXNet symbolic, right? > - “NNVM Compiler now received contributions from AWS, UW and many other > folks in MXNet community.” – agreed it is ramping up, but when you look at > the data, it is clear that it is very early on for NNVM. Looking at the > repo, it has overall 223 commits, 0 releases. Compare it to MXNet with 6136 > commits and 32 releases. It seems to be still early on for NNVM, and for a > more reliable initial implementation building the import on top of MXNet is > easier, faster and safer. MXNet has lots of users already using the > Symbolic API which hopefully mean that is a mature API that is not likely > to have breaking changes or major issues. > > I’m supportive option 1 proposed by Roshani (building serde on top of > MXNet symbolic), but to do it as an encapsulated implementation detail, so > the implementation can be migrated to NNVM or another implementation in the > future, if at that point it seems like the right thing to do. > > Interested in hearing other opinions though… > > Hagay > > On 10/18/17, 14:13, "Tianqi Chen" <[email protected] on behalf of > [email protected]> wrote: > > I am strongly recommending going through the nnvm/top. One major > reason in > here is that the support of nnvm/top layer NOT ONLY mean compatibility > of > model format with onnx. These are the major benefits: > > > - More hardware backends to mxnet, including opencl, metal, Raspberry > Pi, > web browser. These things are automatically enabled by going through > this > layer. In general, we design nnvm/tvm stack to resolve the challenge of > current mxnet's weakness in terms deploying to more hardware backends. > > - More frontend capabilities, nnvm's gluon style IR ingests now from > CoreML, ONNX and in future keras. Supporting those will reduce the > amount > of engineering effort needed. > > - Future compatibility. We all agree that the future being migrated to > gluon's API. NNVM/top tries to look ahead by directly adopting the > symbolic > API to be gluon. > > > I would also like to correct some of the mentioned facts with regard to > nnvm/tvm stack > > 1. Nascent project with few contributors > > NNVM Compiler now received contributions from AWS, UW and many other > folks > in MXNet community. NNVM itself is already being used by MXNet. > MXNet's internal IR is migrating toward gluon, and its final form being > nnvm/top > > 3. Does not support all operators that exist in MXNet Symbolic API > > Neither NNVM/top or onnx support all operators that exist in mxnet > symbolic > API. The end goal here is mainly to make nnvm/top onnx compatible, > which is > a more reasonable goal. > > 4. No CI Pipeline and testcases > > NNVM already contains a compiler contains unittests and ci tested with > integration https://github.com/dmlc/nnvm, with a CI pipline that is > well > tested on CPU and GPU cases for front-ends. > > Tianqi > > > On Wed, Oct 18, 2017 at 1:41 PM, Roshani Nagmote < > [email protected]> > wrote: > > > Hi guys, > > > > > > I am working on supporting ONNX <https://github.com/onnx/onnx> > pre-trained > > models in Apache MXNet and would like to seek your opinion on the > choice of > > implementation. I also have created a GitHub issue > > <https://github.com/apache/incubator-mxnet/issues/8319>. Supporting > ONNX > > in > > MXNet will enable users to move between frameworks with their > models, this > > will also enable MXNet project to be a part of the ONNX open > standard and > > steer the direction of ONNX. > > > > > > For those who don’t know ONNX, ONNX is an open source format for AI > models > > which enables models to be transferred between frameworks. Refer to > > https://github.com/onnx/onnx for more details. > > > > > > To implement the import/export functionality in MXNet, I propose to > expose > > a MXNet python module “serde”(name taken from Apache Hive project) > with the > > following methods supporting different formats: > > > > sym, params = mxnet.serde.import(other_format_file, > other_format=‘onnx’) > > > > other_format_file = mxnet.serde.export(mxnet_sym, mxnet_params, > ‘onnx’) > > > > > > The implementation under the hood can be done in two ways: > > > > > > 1) Implement at the MXNet layer by parsing the ONNX model(in protobuf > > format) and turn into MXNet Symbolic operators and build MXNet model > > directly. Similarly, I can convert the MXNet model to ONNX format at > this > > layer. > > > > > > 2) The DMLC community has released the nnvm/tvm complier and an > > intermediate representation of the models, refer: > > http://www.tvmlang.org/2017/10/06/nnvm/tvm-compiler- > announcement.html > > <http://www.tvmlang.org/2017/10/06/nnvm-compiler-announcement.html> > > > > Based on the conversation on the GitHub issue > > <https://github.com/apache/incubator-mxnet/issues/8319> I opened, Mu > > mentioned that MXNet would use nnvm/tvm as the backend in the future. > > > > > > We could hook into this layer to implement the import/export > functionality. > > nnvm/tvm has ONNX 0.1 version import implemented. > > > > For import, > > > > 1. > > > > I will need to enhance nnvm/tvm’s importer to support ONNX 0.2 > > 2. > > > > Implement nnvm/tvm->mxnet symbolic operators. > > > > For export: > > > > > > 1. > > > > mxnet->nnvm/tvm ( nnvm/tvm provides this implementation already) > > 2. > > > > I will need to Implement nnvm/tvm>onnx. > > > > > > These are the pros and cons I see in the above approaches: > > > > 1. > > > > Import/export at mxnet layer > > > > Pros: > > > > 1. > > > > Stable APIs currently used by users. > > 2. > > > > Larger Apache MXNet community of contributors. > > 3. > > > > CI pipeline to catch bugs. > > 4. > > > > Comparatively less time to implement and put it in the hands of > the > > users. > > > > Cons: > > > > 1. > > > > In the future we may have to reimplement at the nnvm/tvm layer, > in case > > MXNet moves to the nnvm/tvm backend(assuming it will move). > > > > > > > > 1. > > > > Import/export at nnvm/tvm layer > > > > Pros: > > > > 1. > > > > Less engineering work in case mxnet moves to nnvm/tvm > > 2. > > > > nnvm/tvm would become a hub to convert to different formats. > > 3. > > > > nnvm operators are more in parity with mxnet’s gluon APIs this > could be > > useful in case Gluon becomes the only standard that MXNet will > support. > > > > Cons: > > > > 1. > > > > Nascent project with few contributors > > 2. > > > > Does not support all operators that exist in MXNet Symbolic API > > 3. > > > > No CI Pipeline > > 4. > > > > Current Apache MXNet project does not use nnvm/tvm backend > > 5. > > > > mxnet->nnvm/tvm backend needs more testing and user feedback. > > > > > > Any suggestions on both of these approaches? From user's > perspective, this > > will be an implementation detail that is not exposed. > > > > Thanks, > > > > Roshani > > > > > >
