Mousius commented on issue #8717: URL: https://github.com/apache/tvm/issues/8717#issuecomment-976400930
No problem @huanmei9, > 1. What kind of arm device support is_fast_int8_on_arm()? Could you give some examples? If we take a look at the `is_fast_int8_on_arm()` function, it actually relates do the dot product extension: https://github.com/apache/tvm/blob/02fbaf0ed9120a8f95155e63de42459f230584aa/python/tvm/relay/qnn/op/legalizations.py#L350-L353 This function should be `has_dot_product_on_arm()` to be more explicit about the extension it works with. We should use the attributes and architectures rather than specific devices if possible. > 2. Why the operator need to cast to int16 if device neither use_int8_on_arm nor is_fast_int8_on_arm? This is what @manupa-arm was alluding to with: > Yes it is generally applicable for any operator. The reasoning is mentioned in the comment as follows : > `# Advanced SIMD present and no dot product extension` -- This is where we should be legalizing to int16 dtypes. Where-in it's more effective to use `int16` operations if there's no dot product extension available. This is why we should change this code (@Alex-grovety was correct, I referenced dense instead of conv2d previously, both suffer from the same issue I think, apologies for the confusion): https://github.com/apache/tvm/blob/02fbaf0ed9120a8f95155e63de42459f230584aa/python/tvm/relay/qnn/op/legalizations.py#L377-L380 To something similar to: ``` if (is_aarch64_arm() or "+neon" in target.mattr) and not is_fast_int8_on_arm(): return helper_no_fast_int8_hw_legalization(attrs, inputs, types, relay.nn.dense) return helper_change_dtypes_to_be_same(attrs, inputs, types, relay.qnn.op.dense) ``` As you're quite correct, by default casting to int16 is more expensive but with a specific set of extensions it becomes more effective (no dot product available). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
