tqchen commented on a change in pull request #23: URL: https://github.com/apache/tvm-rfcs/pull/23#discussion_r719355235
########## File path: rfcs/0023-adding-annotation-field-to-tir.allocate.md ########## @@ -0,0 +1,195 @@ + +- Feature Name: Adding annotatation field to tir.allocate nodes +- Start Date: 2021-06-01 +- RFC PR: https://github.com/apache/tvm-rfcs/pull/23 +- GitHub Issue: TBD + +# 1. Summary + +This RFC proposes to annotation field tir.allocate nodes. These annotations can be used as auxiliary hint to future transformations. + +# 2. Motivational usecase : pinned memory + +Currently, TVM relies on dynamic (alloc and free style) allocations in runtime to manage the intermediary memory used by operators and the network. This is sometimes not desirable, especially in microTVM. + +The current design of [Unified Static Memory Planner (USMP)](https://github.com/apache/tvm-rfcs/pull/9), enables the user option to provide buffers to place workspace and constant tensors. + +``` + tvmc compile my_model.tflite + --executor=aot + --target=accel,c + --with-workspace-buffer= "name=dtcm;target=c;size=1000" # Here the size is more of a hint/guide provided to USMP + --with-workspace-buffer= "name=sram;target=c,accel" + --with-parameter-buffer= "name=itcm;target=c;size=5000" # Here the size is more of a hint/guide provided to USMP + --with-parameter-buffer= "name=flash;target=c,accel" +``` + +``` + // The User Application + extern const TVMModel my_model; + __attribute__((section( "ITCM" ) const uint8_t my_model_params_1[TVM_MY_MODEL_ITCM_PARAMETER_BUFFER_SIZE] = <param_1_data>; + __attribute__((section( "FLASH" ), aligned( 16 ))) const uint8_t my_model_params_2[TVM_MY_MODEL_FLASH_PARAMETER_BUFFER_SIZE] = <param_2_data>; + __attribute__((section( "DTCM" ) static uint8_t workspace_buffer_1[TVM_MY_MODEL_DTCM_WORKSPACE_BUFFER_SIZE]; + __attribute__((section( "SRAM" ), aligned( 16 ))) static uint8_t workspace_buffer_2[TVM_MY_MODEL_SRAM_WORKSPACE_BUFFER_SIZE]; + + int main(...) { + ... + TVMContext context; + TVMInputs_my_model_1 inputs = {input}; + TVMOutputs_my_model_1 outputs = {output}; + TVMWorkspaces_my_model workspaces = { + .sram = &workspace_buffer_1, + .dtcm = &workspace_buffer_2, + }; + TVMParameters_my_model parameters = { + .flash = &my_model_params_1, + .itcm = &my_model_params_2 + }; + TVMSetWorkspaces(&context, &workspaces); + TVMSetParameters(&context, parameters); + TVMExecute(&my_model, &inputs, &outputs, &context); + } +``` + +Therefore, we'd need a way to represent the association of each of these memories, that the user will pin the buffers (e.g. workspace_buffer_2 to SRAM) to. That infomation should flow closer to allocate nodes in TIR in the compilation passes. + +At the IR, we ll need to associate each allocate node with one (or more) memory pools that it can end up, because the scheduling might be satisfied with placing buffers in any of the memory pools in a given set of memory pools. Therefore, the scheduler might want the memory planners to decide which memory pool to use based on finding the allocation that fit. + + There are broadly two requirements here : + +P1 : Indicate candidate memory pools (a.k.a. PoolInfo Objects -- for further details see [USMP RFC](https://github.com/apache/tvm-rfcs/pull/9)) that each allocate be associated with + +P2 : Indicate the final memory pool the allocate will be pinned on + + +To serve P2, we are proposing to use the existing tag of the storage_scope with 'global.<pool_name>' in the [USMP RFC](https://github.com/apache/tvm-rfcs/pull/9). + +To serve P1, this RFC introduces the addition of the annotations field + +# 3. Guide-level explanation + +This is not particularly a user-facing feature. + +The proposal in this RFC is to add an annotation field to tir.allocate node to be used by compilation passes. + + + # 4. Reference-level explanation + + +To serve P1, we propose to use : + +``` +class AllocateNode : public StmtNode { + public: + /*! \brief The buffer variable. */ + Var buffer_var; + /*! \brief The type of the buffer. */ + DataType dtype; + /*! \brief The extents of the buffer. */ + Array<PrimExpr> extents; + /*! \brief Only allocate buffer when condition is satisfied. */ + PrimExpr condition; + /*! \brief The body to be executed. */ + Stmt body; + /*! \brief If the allocate is scoped global, this field indicates + * which external memories it could be pinned to as a comma seperated Review comment: NOTE: the comments here shall be updated that is independent from the pinned memory. as in the ForNode.annotations -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
