mbs-octoml commented on a change in pull request #10: URL: https://github.com/apache/tvm-rfcs/pull/10#discussion_r714143075
########## File path: rfcs/0010-target-registered-compiler-flow-customisation.md ########## @@ -0,0 +1,186 @@ +- Feature Name: `Target` registered compiler flow customisation +- Start Date: 2021-07-14 +- RFC PR: https://github.com/apache/tvm-rfcs/pull/10 +- GitHub Issue: https://github.com/apache/tvm/issues/8589 + +# Summary +[summary]: #summary + +In order to enable flexibility in how individual targets are lowered and built within TVM, this RFC proposes additional hooks on the `Target` and that the target becomes the central place for such hooks, for example: + +```c++ +using FTVMRelayToTIR = Pass; +using FTVMTIRToRuntime = runtime::TypedPackedFunc<runtime::Module(IRModule, Target)>; + +TVM_REGISTER_TARGET_KIND("cmsisnn", kDLCPU) + .set_attr<FTVMRelayToTIR>("RelayToTIR", CMSISNNLowering) + .set_attr<FTVMTIRToRuntime>("TIRToRuntime", CMSISNNCodeGen); +``` + +This defines two new hooks as attributes on the target, referencing functions registered into the central TVM registry. In similar fashion, external code generators (registered under the `relay.ext.` namespace currently) would be grouped with an appropriate `Target` as well: + +```c++ +using FTVMRelayToRuntime = runtime::TypedPackedFunc<runtime::Module(const Function&)>; +using FTVMConstantUpdater = runtime::TypedPackedFunc<Map<String, runtime::NDArray>(Expr, std::string)>; + +TVM_REGISTER_TARGET_KIND("ethos-n", kDLCPU) + .set_attr<FTVMRelayToRuntime>("RelayToRuntime", EthosNCodeGen) + .set_attr<FTVMConstantUpdater>("UpdateConstants", EthosNConstantUpdater); +``` + +Collecting all targets under the `Target` functionality (as opposed to registering additional `Target`s through the function registry using the namespace `relay.ext.`) and making it clearer which hooks apply to each target. Review comment: nit: '...makes it clearer which...' ########## File path: rfcs/0010-target-registered-compiler-flow-customisation.md ########## @@ -0,0 +1,186 @@ +- Feature Name: `Target` registered compiler flow customisation +- Start Date: 2021-07-14 +- RFC PR: https://github.com/apache/tvm-rfcs/pull/10 +- GitHub Issue: https://github.com/apache/tvm/issues/8589 + +# Summary +[summary]: #summary + +In order to enable flexibility in how individual targets are lowered and built within TVM, this RFC proposes additional hooks on the `Target` and that the target becomes the central place for such hooks, for example: + +```c++ +using FTVMRelayToTIR = Pass; +using FTVMTIRToRuntime = runtime::TypedPackedFunc<runtime::Module(IRModule, Target)>; + +TVM_REGISTER_TARGET_KIND("cmsisnn", kDLCPU) + .set_attr<FTVMRelayToTIR>("RelayToTIR", CMSISNNLowering) + .set_attr<FTVMTIRToRuntime>("TIRToRuntime", CMSISNNCodeGen); +``` + +This defines two new hooks as attributes on the target, referencing functions registered into the central TVM registry. In similar fashion, external code generators (registered under the `relay.ext.` namespace currently) would be grouped with an appropriate `Target` as well: + +```c++ +using FTVMRelayToRuntime = runtime::TypedPackedFunc<runtime::Module(const Function&)>; +using FTVMConstantUpdater = runtime::TypedPackedFunc<Map<String, runtime::NDArray>(Expr, std::string)>; + +TVM_REGISTER_TARGET_KIND("ethos-n", kDLCPU) + .set_attr<FTVMRelayToRuntime>("RelayToRuntime", EthosNCodeGen) + .set_attr<FTVMConstantUpdater>("UpdateConstants", EthosNConstantUpdater); +``` + +Collecting all targets under the `Target` functionality (as opposed to registering additional `Target`s through the function registry using the namespace `relay.ext.`) and making it clearer which hooks apply to each target. + +# Motivation +[motivation]: #motivation + +Currently to introduce an external code generator (otherwise known as [BYOC](https://tvm.apache.org/docs/dev/relay_bring_your_own_codegen.html)), the entire compilation pipeline must be recreated; this is necessary for some targets but in the case of simply re-using existing libraries or introducing a function call to use for an operator it can become more than is necessary; to implement an external code generator requires going directly from Relay to a `runtime::Module` and re-implementing any compiler passes and code generation functionality rather than being able to extend upon the existing compiler infrastructure. + +The generated `runtime::Module` also exists outside of the main graph, meaning it can't be inspected in combination with other operators; this limits the effectiveness of techniques such as memory planning. By introducing the hook `RelayToTIR`, which is similar to the default `LowerTEPass` in that it returns TIR, it can be inspected by the memory planner and other analysis passes that only work at the TIR level. If all that is necessary is transforming into a flat `call_extern` (such is the case for the [CMSIS NN Softmax function](https://github.com/ARM-software/CMSIS_5/blob/develop/CMSIS/NN/Source/SoftmaxFunctions/arm_softmax_s8.c#L86)) then the hook may simply return that TIR to be collected by the host code generation. + +In the more complex case, we still want to take advantage of memory planning by using `RelayToTIR` and inspecting the liveness within the TIR graph, but instead want to generate out more complex calls (such as using the [CMSIS NN Structures](https://github.com/ARM-software/CMSIS_5/blob/def6f800f95661eb3451d317f7d0dde504f6020d/CMSIS/NN/Include/arm_nn_types.h#L81-L90)); the `TIRToRuntime` hook can be used to build our intermediary TIR into a Runtime module similarly to how the existing external code generation works. This allows writing of external code generators that also get the benefits of any intermediary analysis or transformation that TVM offers. Alongside being able to use the analysis passes, code generators can extend from existing host code generators, customising only the generation which is relevant to them and gaining maximum benefit from the existing optimisations made in TVM. + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +As a user, you can pick from additional hooks to bypass certain behaviours of the `Target`: +* `RelayToTIR` - Customize the lowering flow to TIR +* `TIRToRuntime` - Customize code generation into a runtime module from TIR +* `RelayToRuntime` - Full compilation flow from Relay to a runtime module + +To illustrate where the hooks are placed, please refer to the following diagram: + + + +These can be registered on targets using `set_attr`: +``` +TVM_REGISTER_TARGET_KIND("cmsisnn", kDLCPU) + .set_attr<FTVMRelayToTIR>("RelayToTIR", CMSISNNLowering) + .set_attr<FTVMTIRToRuntime>("TIRToRuntime", CMSISNNCodeGen); + +TVM_REGISTER_TARGET_KIND("ethos-n", kDLCPU) + .set_attr<FTVMRelayToRuntime>("RelayToRuntime", EthosNCodeGen) + .set_attr<FTVMConstantUpdater>("UpdateConstants", EthosNConstantUpdater); +``` + +## Relay -> TIR +With this change, this path splits, depending on whether you wanted to generate a full `Module` or introduce some specific TIR nodes into the code generation flow; the addition of the `RelayToTIR` hook allows you to write trivial external TIR generators such as calling out to a third party library: +```c++ +void CallExternalLibraryInTIR(const GlobalVar& new_global_var, const Function& func) { + tir::Buffer x_buffer = tir::decl_buffer({8}, DataType::Float(32), "x"); + tir::Var x_var("x", DataType::Handle()); + + Map<String, ObjectRef> dict_attrs; + dict_attrs.Set("global_symbol", new_global_var->name_hint); + dict_attrs.Set("tir.noalias", Bool(true)); + + Map<tir::Var, tir::Buffer> buffer_map = {{x_var, x_buffer}}; + tir::Stmt body = + tir::Evaluate(tvm::tir::Call(DataType::Int(8), tir::builtin::call_extern(), {x->data})); + + tir::PrimFunc replacement_func = tir::PrimFunc({x_var}, body, VoidType(), + buffer_map, DictAttrs(dict_attrs)); + replacement_func = WithAttr(replacement_func, ::tvm::attr::kTarget, host_target_); + ir_module_->Add(new_global_var, replacement_func); +} +``` +This is then registered on a target: +```c++ +TVM_REGISTER_TARGET_KIND("woofles", kDLCPU) + .set_attr<FTVMRelayToTIR>("RelayToTIR", relay::contrib::woofles::RelayToTIR()); +``` +The signature for this hook is as the same as any other `Pass`, which takes an `IRModule` with `Function`s and returns an `IRModule` with transformed `PrimFunc`s. + Review comment: nit: something like "The registered pass is responsible for both establishing the PrimFunc definitions (with any caching) and rewriting Relay calls to those functions. At this time we feel it's not worth worrying about code sharing between different custom passes". Just to capture the consensus on that. ########## File path: rfcs/0010-target-registered-compiler-flow-customisation.md ########## @@ -0,0 +1,186 @@ +- Feature Name: `Target` registered compiler flow customisation +- Start Date: 2021-07-14 +- RFC PR: https://github.com/apache/tvm-rfcs/pull/10 +- GitHub Issue: https://github.com/apache/tvm/issues/8589 + +# Summary +[summary]: #summary + +In order to enable flexibility in how individual targets are lowered and built within TVM, this RFC proposes additional hooks on the `Target` and that the target becomes the central place for such hooks, for example: + +```c++ +using FTVMRelayToTIR = Pass; +using FTVMTIRToRuntime = runtime::TypedPackedFunc<runtime::Module(IRModule, Target)>; + +TVM_REGISTER_TARGET_KIND("cmsisnn", kDLCPU) + .set_attr<FTVMRelayToTIR>("RelayToTIR", CMSISNNLowering) + .set_attr<FTVMTIRToRuntime>("TIRToRuntime", CMSISNNCodeGen); +``` + +This defines two new hooks as attributes on the target, referencing functions registered into the central TVM registry. In similar fashion, external code generators (registered under the `relay.ext.` namespace currently) would be grouped with an appropriate `Target` as well: + +```c++ +using FTVMRelayToRuntime = runtime::TypedPackedFunc<runtime::Module(const Function&)>; +using FTVMConstantUpdater = runtime::TypedPackedFunc<Map<String, runtime::NDArray>(Expr, std::string)>; + +TVM_REGISTER_TARGET_KIND("ethos-n", kDLCPU) + .set_attr<FTVMRelayToRuntime>("RelayToRuntime", EthosNCodeGen) + .set_attr<FTVMConstantUpdater>("UpdateConstants", EthosNConstantUpdater); +``` + +Collecting all targets under the `Target` functionality (as opposed to registering additional `Target`s through the function registry using the namespace `relay.ext.`) and making it clearer which hooks apply to each target. + +# Motivation +[motivation]: #motivation + +Currently to introduce an external code generator (otherwise known as [BYOC](https://tvm.apache.org/docs/dev/relay_bring_your_own_codegen.html)), the entire compilation pipeline must be recreated; this is necessary for some targets but in the case of simply re-using existing libraries or introducing a function call to use for an operator it can become more than is necessary; to implement an external code generator requires going directly from Relay to a `runtime::Module` and re-implementing any compiler passes and code generation functionality rather than being able to extend upon the existing compiler infrastructure. + +The generated `runtime::Module` also exists outside of the main graph, meaning it can't be inspected in combination with other operators; this limits the effectiveness of techniques such as memory planning. By introducing the hook `RelayToTIR`, which is similar to the default `LowerTEPass` in that it returns TIR, it can be inspected by the memory planner and other analysis passes that only work at the TIR level. If all that is necessary is transforming into a flat `call_extern` (such is the case for the [CMSIS NN Softmax function](https://github.com/ARM-software/CMSIS_5/blob/develop/CMSIS/NN/Source/SoftmaxFunctions/arm_softmax_s8.c#L86)) then the hook may simply return that TIR to be collected by the host code generation. + +In the more complex case, we still want to take advantage of memory planning by using `RelayToTIR` and inspecting the liveness within the TIR graph, but instead want to generate out more complex calls (such as using the [CMSIS NN Structures](https://github.com/ARM-software/CMSIS_5/blob/def6f800f95661eb3451d317f7d0dde504f6020d/CMSIS/NN/Include/arm_nn_types.h#L81-L90)); the `TIRToRuntime` hook can be used to build our intermediary TIR into a Runtime module similarly to how the existing external code generation works. This allows writing of external code generators that also get the benefits of any intermediary analysis or transformation that TVM offers. Alongside being able to use the analysis passes, code generators can extend from existing host code generators, customising only the generation which is relevant to them and gaining maximum benefit from the existing optimisations made in TVM. + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +As a user, you can pick from additional hooks to bypass certain behaviours of the `Target`: +* `RelayToTIR` - Customize the lowering flow to TIR +* `TIRToRuntime` - Customize code generation into a runtime module from TIR +* `RelayToRuntime` - Full compilation flow from Relay to a runtime module + +To illustrate where the hooks are placed, please refer to the following diagram: + + + +These can be registered on targets using `set_attr`: +``` +TVM_REGISTER_TARGET_KIND("cmsisnn", kDLCPU) + .set_attr<FTVMRelayToTIR>("RelayToTIR", CMSISNNLowering) + .set_attr<FTVMTIRToRuntime>("TIRToRuntime", CMSISNNCodeGen); + +TVM_REGISTER_TARGET_KIND("ethos-n", kDLCPU) + .set_attr<FTVMRelayToRuntime>("RelayToRuntime", EthosNCodeGen) + .set_attr<FTVMConstantUpdater>("UpdateConstants", EthosNConstantUpdater); +``` + +## Relay -> TIR +With this change, this path splits, depending on whether you wanted to generate a full `Module` or introduce some specific TIR nodes into the code generation flow; the addition of the `RelayToTIR` hook allows you to write trivial external TIR generators such as calling out to a third party library: +```c++ +void CallExternalLibraryInTIR(const GlobalVar& new_global_var, const Function& func) { + tir::Buffer x_buffer = tir::decl_buffer({8}, DataType::Float(32), "x"); + tir::Var x_var("x", DataType::Handle()); + + Map<String, ObjectRef> dict_attrs; + dict_attrs.Set("global_symbol", new_global_var->name_hint); + dict_attrs.Set("tir.noalias", Bool(true)); + + Map<tir::Var, tir::Buffer> buffer_map = {{x_var, x_buffer}}; + tir::Stmt body = + tir::Evaluate(tvm::tir::Call(DataType::Int(8), tir::builtin::call_extern(), {x->data})); + + tir::PrimFunc replacement_func = tir::PrimFunc({x_var}, body, VoidType(), + buffer_map, DictAttrs(dict_attrs)); + replacement_func = WithAttr(replacement_func, ::tvm::attr::kTarget, host_target_); + ir_module_->Add(new_global_var, replacement_func); +} +``` +This is then registered on a target: +```c++ +TVM_REGISTER_TARGET_KIND("woofles", kDLCPU) + .set_attr<FTVMRelayToTIR>("RelayToTIR", relay::contrib::woofles::RelayToTIR()); +``` +The signature for this hook is as the same as any other `Pass`, which takes an `IRModule` with `Function`s and returns an `IRModule` with transformed `PrimFunc`s. + +## TIR -> Runtime +Extending from the above, a second hook is introduced to do further transformations from TIR -> Runtime named `TIRToRuntime`, this bypasses the default `target.build.X` and instead uses the registered `TIRToRuntime` build: +```c++ +runtime::Module BuildWooflesHost(IRModule mod, Target target) { Review comment: nit: Make clear the IRModule can still be cross-target, right? Ie all of these hooks are responsible for doing the 'just look at defns for my target' filtering? ########## File path: rfcs/0010-target-registered-compiler-flow-customisation.md ########## @@ -0,0 +1,186 @@ +- Feature Name: `Target` registered compiler flow customisation +- Start Date: 2021-07-14 +- RFC PR: https://github.com/apache/tvm-rfcs/pull/10 +- GitHub Issue: https://github.com/apache/tvm/issues/8589 + +# Summary +[summary]: #summary + +In order to enable flexibility in how individual targets are lowered and built within TVM, this RFC proposes additional hooks on the `Target` and that the target becomes the central place for such hooks, for example: + +```c++ +using FTVMRelayToTIR = Pass; +using FTVMTIRToRuntime = runtime::TypedPackedFunc<runtime::Module(IRModule, Target)>; + +TVM_REGISTER_TARGET_KIND("cmsisnn", kDLCPU) + .set_attr<FTVMRelayToTIR>("RelayToTIR", CMSISNNLowering) + .set_attr<FTVMTIRToRuntime>("TIRToRuntime", CMSISNNCodeGen); +``` + +This defines two new hooks as attributes on the target, referencing functions registered into the central TVM registry. In similar fashion, external code generators (registered under the `relay.ext.` namespace currently) would be grouped with an appropriate `Target` as well: + +```c++ +using FTVMRelayToRuntime = runtime::TypedPackedFunc<runtime::Module(const Function&)>; +using FTVMConstantUpdater = runtime::TypedPackedFunc<Map<String, runtime::NDArray>(Expr, std::string)>; + +TVM_REGISTER_TARGET_KIND("ethos-n", kDLCPU) + .set_attr<FTVMRelayToRuntime>("RelayToRuntime", EthosNCodeGen) + .set_attr<FTVMConstantUpdater>("UpdateConstants", EthosNConstantUpdater); +``` + +Collecting all targets under the `Target` functionality (as opposed to registering additional `Target`s through the function registry using the namespace `relay.ext.`) and making it clearer which hooks apply to each target. + +# Motivation +[motivation]: #motivation + +Currently to introduce an external code generator (otherwise known as [BYOC](https://tvm.apache.org/docs/dev/relay_bring_your_own_codegen.html)), the entire compilation pipeline must be recreated; this is necessary for some targets but in the case of simply re-using existing libraries or introducing a function call to use for an operator it can become more than is necessary; to implement an external code generator requires going directly from Relay to a `runtime::Module` and re-implementing any compiler passes and code generation functionality rather than being able to extend upon the existing compiler infrastructure. Review comment: nit: May be worth a pithy opening sentence saying we want to make BOYC more modular: instead of going from Relay to Module in one big step you can a) break it into phases and b) make use of existing transformations between phases. Or at least that's the dumbed-down way I've been thinking of it. ########## File path: rfcs/0010-target-registered-compiler-flow-customisation.md ########## @@ -0,0 +1,186 @@ +- Feature Name: `Target` registered compiler flow customisation +- Start Date: 2021-07-14 +- RFC PR: https://github.com/apache/tvm-rfcs/pull/10 +- GitHub Issue: https://github.com/apache/tvm/issues/8589 + +# Summary +[summary]: #summary + +In order to enable flexibility in how individual targets are lowered and built within TVM, this RFC proposes additional hooks on the `Target` and that the target becomes the central place for such hooks, for example: + +```c++ +using FTVMRelayToTIR = Pass; +using FTVMTIRToRuntime = runtime::TypedPackedFunc<runtime::Module(IRModule, Target)>; + +TVM_REGISTER_TARGET_KIND("cmsisnn", kDLCPU) + .set_attr<FTVMRelayToTIR>("RelayToTIR", CMSISNNLowering) + .set_attr<FTVMTIRToRuntime>("TIRToRuntime", CMSISNNCodeGen); +``` + +This defines two new hooks as attributes on the target, referencing functions registered into the central TVM registry. In similar fashion, external code generators (registered under the `relay.ext.` namespace currently) would be grouped with an appropriate `Target` as well: + +```c++ +using FTVMRelayToRuntime = runtime::TypedPackedFunc<runtime::Module(const Function&)>; +using FTVMConstantUpdater = runtime::TypedPackedFunc<Map<String, runtime::NDArray>(Expr, std::string)>; + +TVM_REGISTER_TARGET_KIND("ethos-n", kDLCPU) + .set_attr<FTVMRelayToRuntime>("RelayToRuntime", EthosNCodeGen) + .set_attr<FTVMConstantUpdater>("UpdateConstants", EthosNConstantUpdater); +``` + +Collecting all targets under the `Target` functionality (as opposed to registering additional `Target`s through the function registry using the namespace `relay.ext.`) and making it clearer which hooks apply to each target. + +# Motivation +[motivation]: #motivation + +Currently to introduce an external code generator (otherwise known as [BYOC](https://tvm.apache.org/docs/dev/relay_bring_your_own_codegen.html)), the entire compilation pipeline must be recreated; this is necessary for some targets but in the case of simply re-using existing libraries or introducing a function call to use for an operator it can become more than is necessary; to implement an external code generator requires going directly from Relay to a `runtime::Module` and re-implementing any compiler passes and code generation functionality rather than being able to extend upon the existing compiler infrastructure. + +The generated `runtime::Module` also exists outside of the main graph, meaning it can't be inspected in combination with other operators; this limits the effectiveness of techniques such as memory planning. By introducing the hook `RelayToTIR`, which is similar to the default `LowerTEPass` in that it returns TIR, it can be inspected by the memory planner and other analysis passes that only work at the TIR level. If all that is necessary is transforming into a flat `call_extern` (such is the case for the [CMSIS NN Softmax function](https://github.com/ARM-software/CMSIS_5/blob/develop/CMSIS/NN/Source/SoftmaxFunctions/arm_softmax_s8.c#L86)) then the hook may simply return that TIR to be collected by the host code generation. + +In the more complex case, we still want to take advantage of memory planning by using `RelayToTIR` and inspecting the liveness within the TIR graph, but instead want to generate out more complex calls (such as using the [CMSIS NN Structures](https://github.com/ARM-software/CMSIS_5/blob/def6f800f95661eb3451d317f7d0dde504f6020d/CMSIS/NN/Include/arm_nn_types.h#L81-L90)); the `TIRToRuntime` hook can be used to build our intermediary TIR into a Runtime module similarly to how the existing external code generation works. This allows writing of external code generators that also get the benefits of any intermediary analysis or transformation that TVM offers. Alongside being able to use the analysis passes, code generators can extend from existing host code generators, customising only the generation which is relevant to them and gaining maximum benefit from the existing optimisations made in TVM. + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +As a user, you can pick from additional hooks to bypass certain behaviours of the `Target`: +* `RelayToTIR` - Customize the lowering flow to TIR +* `TIRToRuntime` - Customize code generation into a runtime module from TIR +* `RelayToRuntime` - Full compilation flow from Relay to a runtime module + +To illustrate where the hooks are placed, please refer to the following diagram: + + + +These can be registered on targets using `set_attr`: +``` +TVM_REGISTER_TARGET_KIND("cmsisnn", kDLCPU) + .set_attr<FTVMRelayToTIR>("RelayToTIR", CMSISNNLowering) + .set_attr<FTVMTIRToRuntime>("TIRToRuntime", CMSISNNCodeGen); + +TVM_REGISTER_TARGET_KIND("ethos-n", kDLCPU) + .set_attr<FTVMRelayToRuntime>("RelayToRuntime", EthosNCodeGen) + .set_attr<FTVMConstantUpdater>("UpdateConstants", EthosNConstantUpdater); +``` + +## Relay -> TIR +With this change, this path splits, depending on whether you wanted to generate a full `Module` or introduce some specific TIR nodes into the code generation flow; the addition of the `RelayToTIR` hook allows you to write trivial external TIR generators such as calling out to a third party library: Review comment: nit: I'd spell out that by this point every Function has a kTarget annotation thanks to AnnotateTarget, and that IRModule will hold all defns for all targets right up until the transition to Module. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
