ccjoechou commented on a change in pull request #48: URL: https://github.com/apache/tvm-rfcs/pull/48#discussion_r787275464
########## File path: rfcs/0048-BYOC-Marvell-ML-accelerator-integration.md ########## @@ -0,0 +1,547 @@ +- Feature Name: (fill me in with a unique identifier, `my_awesome_feature`) +- Start Date: (fill me in with today's date, YYYY-MM-DD) +- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000) +- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000) +- GitHub pre-RFC PR: [apache/tvm-PR-9730](https://github.com/apache/tvm/pull/9730) +- GitHub pre-RFC discussion: [BYOC-Marvell](https://discuss.tvm.apache.org/t/pre-rfc-byoc-marvell-ml-ai-accelerator-integration/11691) + +# Summary +[summary]: #summary + +Integrate Marvell’s ML/AI accelerator with TVM BYOC framework in order to bring the TVM ecosystem to Marvell customers. + +# Motivation +[motivation]: #motivation + +Marvell MLIP is an ML/AI inference accelerator and is embedded on our ARM Neoverse N2-based OCTEON 10 processor. + We are building an easy-to-use, open, software suite for our customers by integrating and utilizing TVM so that + we can bring TVM capability and experience to our customers. + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +Based on what Marvell ML/AI inference accelerator does the best, a given pre-trained network model +will be applied to a TVM-Mrvl-BYOC AOT compilation and code-gen flow as illustrated in steps below. + +STEP (1) Run TVM-Mrvl-BYOC AOT ML Frontend Compilation and Mrvl-BYOC code-gen. The steps involved in this are: + +* Load pre-trained network into TVM IR graph + +* Do Marvell-specific layout conversions to transform IR graph in order to meet requirements of the accelerator + +* Do Marvell-specific composite-merging/fusing to transform IR graph in order to utilize available HW capability + in the accelerator + +* Do additional Marvell-specific transform pass(es) to further optimize IR graph + +* Partition IR graph into one or more for-accelerator Mrvl subgraphs and/or one or more for-TVM-target non-Mrvl + (e.g., ARMv9) subgraphs + * These subgraphs cover the whole pre-trained network + * For-accelerator Mrvl subgraph here means & contains connected, composite-fused Call nodes (let's call this sub-graph A) + as in the given IR graph. A composite-merged Call node can be, for instance, fused from this sequence of IR call nodes: + conv2d + add + batch_norm + tuple.getitem(0) + relu + * For the first Marvell-BYOC revision, at most one for-accelerator Mrvl subgraph and at most one for-TVM-target + non-Mrvl subgraph (let's call this sub-graph B) can be identified; plus, the for-accelerator Mrvl subgraph can + only use input tensor(s) of given pre-trained network as its subgraph’s input tensors + +* Do code-gen step for each for-accelerator Mrvl subgraph: + * Marvell-BYOC-specific attributes are introduced for each composite-merged/fused Call node so that a Nodes-JSON + file and a Constants-JSON file are produced for the Mrvl subgraph + +STEP (2) Run Mrvl-ML/AI Backend Compiler to generate model binary for each Mrvl subgraph + +* The Mrvl-ML/AI backend compiler will be distributed as an executable in the OCTEON SDK; and it can be used to read + in Nodes-JSON and Constants-JSON files of each Mrvl subgraph as input meta-data in order to generate final instructions, + in model binary file + +* Note: Mrvl-ML/AI backend compiler, which does accelerator-specific optimization and code generation, is not included Review comment: for this BYOC-Marvell RFC, the POC PR codebase only contains code to generate JSON meta files. We have up-streamed our test_mrvl test suite but only contains JSON codegen. In our next RFC, we will provide runtime & driver hookups. We are working on a Marvell backend package with Marvell backend code-gen and Marvell software simulator, which mimics a cycle-approximate Marvell HW accelerator. This package can become available later for external usage. Currently, we are having problems run TVM rust/cargo and can’t find useful document to debug issues – plus, tvm-build is owned by OctoML (not GitHub TVM, right?) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
