manupa-arm commented on a change in pull request #23: URL: https://github.com/apache/tvm-rfcs/pull/23#discussion_r714771648
########## File path: rfcs/0023-adding-annotation-field-to-tir.allocate.md ########## @@ -0,0 +1,201 @@ + +- Feature Name: Adding annotatation field to tir.allocate nodes +- Start Date: 2021-06-01 +- RFC PR: https://github.com/apache/tvm-rfcs/pull/23 +- GitHub Issue: TBD + +# 1. Summary + +This RFC proposes to annotation field tir.allocate nodes. These annotations can be used as auxiliary hint to future transformations. + +# 2. Motivational usecase : pinned memory + +Currently, TVM relies on dynamic (alloc and free style) allocations in runtime to manage the intermediary memory used by operators and the network. This is sometimes not desirable, especially in microTVM. + +The current design of [Unified Static Memory Planner (USMP)](https://github.com/apache/tvm-rfcs/pull/9), enables the user option to provide buffers to place workspace and constant tensors. + +``` + tvmc compile my_model.tflite + --executor=aot + --target=accel,c + --with-workspace-buffer= "name=dtcm;target=c;size=1000" # Here the size is more of a hint/guide provided to USMP + --with-workspace-buffer= "name=sram;target=c,accel" + --with-parameter-buffer= "name=itcm;target=c;size=5000" # Here the size is more of a hint/guide provided to USMP + --with-parameter-buffer= "name=flash;target=c,accel" +``` + +``` + // The User Application + extern const TVMModel my_model; + __attribute__((section( "ITCM" ) const uint8_t my_model_params_1[TVM_MY_MODEL_ITCM_PARAMETER_BUFFER_SIZE] = <param_1_data>; + __attribute__((section( "FLASH" ), aligned( 16 ))) const uint8_t my_model_params_2[TVM_MY_MODEL_FLASH_PARAMETER_BUFFER_SIZE] = <param_2_data>; + __attribute__((section( "DTCM" ) static uint8_t workspace_buffer_1[TVM_MY_MODEL_DTCM_WORKSPACE_BUFFER_SIZE]; + __attribute__((section( "SRAM" ), aligned( 16 ))) static uint8_t workspace_buffer_2[TVM_MY_MODEL_SRAM_WORKSPACE_BUFFER_SIZE]; + + int main(...) { + ... + TVMContext context; + TVMInputs_my_model_1 inputs = {input}; + TVMOutputs_my_model_1 outputs = {output}; + TVMWorkspaces_my_model workspaces = { + .sram = &workspace_buffer_1, + .dtcm = &workspace_buffer_2, + }; + TVMParameters_my_model parameters = { + .flash = &my_model_params_1, + .itcm = &my_model_params_2 + }; + TVMSetWorkspaces(&context, &workspaces); + TVMSetParameters(&context, parameters); + TVMExecute(&my_model, &inputs, &outputs, &context); + } +``` + +Therefore, we'd need a way to represent the association of each of these memories, that the user will pin the buffers to, closer to allocate nodes in TIR. + +At the IR, we ll need to associate each allocate node with one (or more) memories that it can end up, because the scheduling might be satisfied with placing buffers in any of the memories in a given set of memories. Therefore, the scheduler might want the memory planners to decide which memory to use based on finding the allocation that fit. + + There are broadly two requirements here : + +P1 : Indicate candidate memories (a.k.a. Pools) that each allocate be associated with + +P2 : Indicate the final memory the allocate will be pinned on + + +To serve P2, we propose to use the existing tag of the storage_scope with 'global-<memory_name>'. + +``` +struct StorageScope { + /*! \brief The rank of the storage */ + StorageRank rank{StorageRank::kGlobal}; + /*! \brief tag for special purpose memory. */ + std::string tag; +``` + +To serve P1, this RFC introduces the addition of the annotations field + +# 3. Guide-level explanation + +This is not particularly a user-facing feature. Review comment: I think we should be careful in what we expose to the user and I'd rather keep them just exposed to passes until the feature gets stable and mature. If and when we decide to open this upto the user, we could maybe do the following: convert Object possibly introducing "fromString" and "toString" methods I suppose. If and when we decide to add that feature (for e.g. for ForNode s) we could do the same here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
