Dear MXNet Community,
A couple of weeks ago I sent out my custom operator profiling enhancement project RFC (https://github.com/apache/incubator-mxnet/issues/15241) to the community and got positive feedbacks, and now, the PR ( https://github.com/apache/incubator-mxnet/pull/15210) has been merged. A new section on profiling custom operators has also been added to the profiler tutorial here: https://mxnet.incubator.apache.org/versions/master/tutorials/python/profiler.html#profiling-custom-operators. With that said, here I will have a brief introduction to the changes: Before, MXNet allowed user-defined custom operators, but their profiling was not well supported. More specifically, the custom operator events as we see in the dump file were way longer than the actual execution time of the operators. Also, custom operators consist of pure python code and NDArray operators (“sub-operators”), but at a particular point in a custom operator event, we were not able to tell which ones were executing. To allow for a more informative custom operator profiling, my project helps break a single custom operator call into several events: now, we have individual events for both the “pure python” code, namely the forward() and backward() functions, and the “sub-operators” , which are the NDArray operators we call within forward() and backward() of the custom operator. All the new, fine-grained events will fall under a new domain/category called “Custom Operator”. Also, each event there will have a prefix in its name, which is the name we give to the custom operator. Below is a sample dump file (opened in chome://tracing): [image: /var/folders/fb/wknbw2r95zs5vkwrz9r80bzjsgk_ng/T/com.microsoft.Outlook/WebArchiveCopyPasteTempFiles/cidimage001.png@01D52DA3.0A2F7860] Note that: we need to make sure “profile_imperative” is set to True even when we are using custom operators in symbolic mode. The reason is that within custom operators, even in symbolic mode, sub-operators (and pure python code) are still called imperatively. Should there be any issue using this new feature, please feel free to tag me. Additionally, any feedback is welcome! Thanks, Zhaoqi
