samskalicky commented on issue #19521: URL: https://github.com/apache/incubator-mxnet/issues/19521#issuecomment-751914619
Hi @AlexanderSerov great question, let me try and break it down. > In our design of androidnn we approach a situation when we need to pass devices to androidnn backend. Usually, other backends (mkl, tensort) get a device through Context. The problem is context support limited list of devices (CPU,GPU). On the other hand, androidnn support other set of devices (cpu, gpu, npu...) with indexes specific to android and acquired via android ANeuralNetworks_getDevice api. Contexts in MXNet are chosen by users with the expectation that they can run their whole model with any particular context. The MXNet community has worked hard to maintain parity between CPU and GPU contexts so that the majority of models can be executed on either context successfully. A Context in MXNet needs to be able to support executing all currently supported operators. How operators are implemented (custom C++, BLAS libraries, or custom NN libraries like MKL) is not visible at the context level, rather these are build-time configurations. For example, an MXNet user will use the same CPU context whether or not they use MKL or OpenBLAS as the BLAS library or whether they choose to use MKLDNN/oneDNN. We consider this a huge usability feature in MXNet, rather than having many contexts to enable each feature. Most users will find the build config that works best for them and stick to that. Having a single build with all those features enabled is not what most users want, inevitably they end up trying to minimize space on disk/device memory and try and reduce the size of the MXNet binary (like our previous discussion above). > So we need custom context and we have choices: > 1. Modify existing Context by adding additional fields and defining a preprocessor flag MXNET_USE_ANDROIDNN in CMake. So if user pass USE_ANDROIDNN option to CMake he will use a custom context. This solution motivated by the fact that if there is a structure for passing devices - we should use it. Previous backends feel comfortable with provided set of devices, now, it's time to add support for new devices. In general adding new build flags to enable custom backends in MXNet is acceptable, especially in the case where the flag would only be enabled for a particular platform (ie. ARM or Android). Adding support for new backends that would be generally applicable for all CPU types requires much more careful consideration and testing. > 2. The second option is to pass all custom options, including device name and id, through MXOptimizeForBackend api which support options_map which was designed for passing custom options to backend and we can use it by passing all custom info required. Then use it when partition graph by adding a custom device to each subgraph as node attribute. Further, based on attribute, we will create a model in backend for a device based on this field. Amazon is using this 2nd option for its implementation of custom backends for the Inferentia and Elastic Inference Accelerator (EIA) devices, and also for its integration of TVM compilation for MXNet models in the SageMaker Neo service. This is the preferable way to get started. It will allow you to build your custom backend, easily stay current with MXNet versions (upgrading between versions with MXNet extensions is way easier than upgrading a custom fork of the whole MXNet codebase), and simplify your distribution of your custom backend with MXNet. The MXNet community is currently reconsidering how MXNet has been architected for a variety of reasons (licensing issues, maintainability, etc) and looking to make the codebase more modular. So having a module backend is future-proofing your efforts as well. And it doesnt limit your contribution either, if your custom backend becomes popular you can always start a discussion on making your custom backend a default part of the MXNet codebase in the future. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
