Hi dev@ I have observed some architectural limitations on MXNet's architecture that would be beneficial to address in future releases. For example during calculation of higher order gradients it would be needed to access the graph and shape information from FGradient function to be able to do some operations in symbolic.
There's also some other activities such as GPU pointwise fusion <https://cwiki.apache.org/confluence/display/MXNET/GPU+Pointwise+fusion> which also need advanced transformations. I would suggest we should collect ideas and requirements in the wiki to have an overview of the scope to make informed decisions when the time comes to make these architectural changes. Maybe relay solve all of these problems? Would be in any case good to have requirements in any case. Any thoughts on this?
