leezu commented on issue #18699: URL: https://github.com/apache/incubator-mxnet/issues/18699#issuecomment-661242552
@mjdenkowski > This benchmark was run using a trained Sockeye model. This is a full sequence-to-sequence model with HybridBlocks glued together by Python code. We're running inference (translating a test set with beam search), so there shouldn't be any backward operations. We can share a model with input data and a run script if that will help with debugging. Do you run multiple inference passes in a single run (multiple forward) or only a single one? > Alternatively, do you have an idea of what good tests would be for determining frontend vs backend speed regressions or multi-API overhead? The commit in question contains both changes in the Python frontend and in the C++ backend. To verify which part introduces an overhead, I recommend to partially apply the commit in question, discarding all Python changes and only applying the C++ changes. In more detail, for the Python frontend, the following changes in the commit of question may introduce a overhead for your use-case: - `if self.hybrid_forward.__func__ is not HybridBlock.hybrid_forward:` checks which return `True` and thus trigger the MXNet 1 compatible code-path. You can refer to the `block.py` file. - C API call `MXNDArraySetDeferredComputeVariable` in `parameter.py` when getting a parameter data array. - Call `op.cast` instead of `op.empty` and `op.copyto` for `array.astype` implementation. See `ndarray.py Note that these changes do not affect the `1.x` branch and your code will stop working on the `master` branch the closer we get to MXNet 2 release due to API changes (https://github.com/apache/incubator-mxnet/issues/17676). Most importantly, the existing ndarray operators will be removed in favor of the numpy operators. Your code currently only runs due to convenience backwards compatibility provided during the early stages of MXNet 2 development, which will be removed before the release. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
