areusch commented on a change in pull request #8: URL: https://github.com/apache/tvm-rfcs/pull/8#discussion_r676872571
########## File path: rfcs/0008-microtvm-project-api.md ########## @@ -0,0 +1,509 @@ +- Feature Name: microtvm_project_api +- Start Date: 2020-06-09 +- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000) +- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000) + +# Summary +[summary]: #summary + +This RFC describes how TVM integrates with build systems for unconventional platforms, such as those +for microcontrollers and for other bare-metal scenarios. + +# Motivation +[motivation]: #motivation + +Though TVM's primary goal is generating code to implement models from a high-level description, +there are several reasons why a user might want to interact with a platform's build system through +TVM: + +1. To perform autotuning. TVM's internal operator implementations are merely templates and rely on + an automatic search process to arrive at a fast configuration for the template on a given + platform. This search process requires that TVM iteratively build and time code on the platform. +2. To perform remote model execution. A user may wish to try several different models or schedules + on a different platform without rewriting the platform-specific code. Users can do this by + building generated model code against a generic implementation of the microTVM RPC server. +3. To debug model execution remotely. Some aspects of model execution are easy to debug using a + platform-specific debugger; however, some things, such as analyzing intermediate tensor values, + are more easily accomplished with TVM-specific tooling. By leveraging the generic microTVM + RPC server used in (2), TVM can provide such tooling in a platform-agnostic way. + +TVM currently supports these use cases through a set of interfaces: +1. `tvm.micro.Compiler`: used to produce binary and library artifacts. +2. `tvm.micro.Flasher`: used to program attached hardware +3. `tvm.micro.Transport`: used to communicate with on-device microTVM RPC server + +Thus far, implementations of these interfaces have been made for Zephyr, mBED OS, and for +simulated hardware using a POSIX subprocess. The latter two interfaces have proven to be a +relatively good fit; however, `tvm.micro.Compiler` is difficult to implement because it attempts +to replicate a platform's build system in TVM. TVM does not want to incorporate platform-specific +build logic into its codebase. + +This proposal unifies these three interfaces to form a "Project-level" interface, recognizing that +it's typical to interact with unconventional platforms and their build systems at this level. It +simplifies the `Compiler` interaction into a project-level Build, and adds an explicit +`generate_project` method to the interface. These changes remove the need to build components +and drive the link process from TVM. + +As a goal, this proposal aims to allow for the same use cases as are currently supported with these +improvements: + +1. Integrating more naturally with build systems typical of embedded platforms. +2. Allowing TVM to automatically generate projects platforms to define automated scripts to build projects + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +TVM can interact with platform SDKs via its **Project API**. Such SDKs are common when working with +non-traditional OS platforms, such as those commonly used in embedded systems (e.g. Arduino, Zephyr, +iOS). Given a platform-specific implementation of this Project API, TVM can: +1. Generate projects that integrate implemented TVM models with generic platform runtime componeents +2. Build those generated projects +3. Program attached hardware +4. Drive remote model execution via the TVM RPC Server interface. + +This last capability means that TVM can drive autotuning, remotely perform model inference, and debug +models on non-traditional OS such as Arduino, Zephyr, and mobile platforms such as iOS and Android. + +To provide support for a platform, a **template project** is first defined. Template projects are +expected to exist entirely inside a directory and are identified to TVM by the path to the directory +locally. Template projects may live either inside the TVM repository (when they can be included in the +TVM CI) or in other version control repositories. The template project contains at minium an +implementation of the **Project API** inside an executable program known as the +**TVM Project API Server**. + +To begin working with a particular platform's Project API implementation, the user supplies TVM with +the path to the top-level directory. TVM launches an instance the Project API Server (found at a +standard location in that directory). TVM communicates with the Project API Server using JSON-RPC +over standard OS pipes. + +TVM supplies generated code to the Project API Server using [Model Library Format](0001-model-library-format.md). + +Below is a survey of example workflows used with the Project API Server: + +## Generating a project + +1. The user imports a model into TVM and builds it using `tvm.relay.build`. +2. The user supplies TVM with the path to the template project and a path to a non-existent + directory where the generated project should live. +3. TVM launches a Project API server in the template project. +4. TVM verifies that the template project is indeed a template by invoking the Project API server + `server_info_query` method. +5. TVM invokes the Project API server `generate_project` method to generate the new project. + +## Building and Flashing + +1. The user follows the steps under [Generating a project](#generating-a-project). +2. TVM expects the Project API server to copy itself to the generated project. It launches a + Project API server in the generated project directory. +3. TVM verifies that the generated project is not a template by invoking the Project API server + `server_info_query` method. This method also returns `options` that can be used to customize + the build. +4. TVM invokes the Project API server `build` method to build the project. +5. TVM invokes the Project API server `flash` method to program the attached device. The + `options` can be used to specify a device serial number. + +## Host-driven model inference + +1. The user follows the steps under [Generating a project](#generating-a-project). +2. TVM invokes the Project API server `connect_transport` method to connect to the remote on-device + microTVM RPC server. +3. The microTVM RPC server is attached to a traditional TVM RPC session on the host device. +4. TVM drives inference on-device using the traditional TVM RPC methods. The Project API server + methods `read_transport` and `write_transport` are used to receive and send data. +5. When the inference session is over, TVM invokes the Project API server method `close_transport` + to release any underlying I/O resources, and terminates the Project API server. + +## AutoTVM + +1. The user supplies a kernel to the AutoTVM tuner for search-based optimization. +2. AutoTVM generates a set of task configurations, instantiates each task, and then invokes + `tvm.build` to produce a `Module` for each instantiated task. +3. AutoTVM produces a Model Library Format archive from the `Module` for each instantiated task. +4. AutoTVM passes the Model Library Format archive to the AutoTVM `runner`. Project API overrides the + traditional AutoTVM runner by providing a [`module_loader`](#module-loader). The microTVM + `module_loader` connects to a _supervisor_ TVM RPC server which carries out the microTVM project build + as part of the TVM RPC `session_constructor`. The following steps occur in the + `session_constructor`: + + 1. The Model Library Format tar is uploaded to the supervisor. + 2. The user supplies a path, on the supervisor, to a template project. + 3. The supervisor `session_constructor` performs the steps under + [Building and Flashing](#building-and-flashing). + 4. The supervisor `session_constructor` invokes the Project API server `connect_transport` method + to connect to the remote device. The session constructor registers a traditional TVM RPC + session on the supervisor, and this session is also used by the AutoTVM runner due to the + `session_constructor` mechanism. + +5. The AutoTVM runner measures runtime as normal. +6. The AutoTVM runner disconnects the session, closing the Project API server on the supervisor. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +## Project API implementation + +The Project API is a Remote Procedure Call (RPC)-type mechanism implemented using +[JSON-RPC](https://www.jsonrpc.org/specification). The client and server are implemented in +`python/tvm/micro/project_api`. Tests are implemented in +`tests/python/unittest/test_micro_project_api.py`. + +## Project API interface + +The functions that need to be implemented as part of a Project API server are defined on the +`ProjectAPIHandler` class in `python/tvm/micro/project_api/server.py`: + +``` +class ProjectAPIHandler(metaclass=abc.ABCMeta): + + @abc.abstractmethod + def server_info_query(self) -> ServerInfo: Review comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
