gromero commented on code in PR #98: URL: https://github.com/apache/tvm-rfcs/pull/98#discussion_r1123874703
########## rfcs/0098_on_device_testing.md: ########## @@ -0,0 +1,97 @@ +- Feature Name: On-Device Testing in TVM CI +- Start Date: 2023-01-24 +- RFC PR: [apache/tvm-rfcs#0098](https://github.com/apache/tvm-rfcs/pull/0098) +- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000) +- Authors: [Mehrdad Hessar](https://github.com/mehrdadh), [David Riazati](https://github.com/driazati) +# Summary +[summary]: #summary + +This RFC describes the approach and challenges to add non-merge-blocking hardware testing in TVM CI. + +# Motivation +[motivation]: #motivation + +Testing is a major part of any open source project to show its stability to the users and companies who are adopting the project. More than 700 contributors are involved with TVM who works at various companies with different needs/interests in TVM. This means the demand for thorough testing is increasing every day. At the time of writing, TVM tests generally run on the hardware targets when that hardware is available in the cloud (for example, x86 CPU, i386, GPU and AArch64). In addition, TVM has supports hardware targets that are not available in the cloud, such as embedded devices supported by microTVM or the Hexagon DSP. The TVM CI cannot currently test code on those hardware as part of its CI, leaving a gap in testing. + +It is possible for TVM to include on-device tests for these non-cloud devices in its CI. However, because they are not widely available to use in cloud services, blocking PR merges over failures in those tests could impose an undue burden on contributors who don’t have access to that hardware. In that hypothetical world, all contributors would, at some point, need to find a way to debug those tests on such non-cloud hardware, even if they didn’t have access to it. + +This does not mean TVM community cannot still run tests on these hardware, either as part of CI in a non-merge-blocking way or against `main` at an e.g. nightly or post-merge cadence. This RFC aims to find a way for TVM community members with access to those special hardware to be able to expand coverage of TVM CI in an advisory capacity by adding instances of their hardware to TVM CI. + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +We explain this section by following the simplest case of a hardware-in-loop CI testing which is nightly regression testing. Anyone in the community can run nightly regression tests and provide the results to the community. We expect hardware vendors to be one of the parties primarily interested in having nightly testing on their hardware targets that are supported in TVM, but this document refers to anyone running a test as Device Test Maintainers. + +## Test Procedure +There is a minimal set of requirements that TVM community expects Device Test Maintainers to follow. To add nightly tests to TVM, Device Test Maintainers should implement automation that performs the following steps: +1. **Lookup nightly SHA for testing.** To ensure that results from disparate nightly test suites can be compares, an automated nightly process chooses a TVM sha1 which everyone should use. The bot will merge the new daily commits on main branch to TVM `nightly` branch ([PR13564](https://github.com/apache/tvm/pull/13564) implemented this) at 9:00PM PST. Device Test Maintainers should use the sha1 from this `nightly` branch for testing so we have consistent results across multiple CIs. + +2. **Testing.** At a minimum, Device Test Maintainers should re-run any simulated integration tests ordinarily ran in TVM’s CI on real hardware targets. In addition, they are welcome to bring more tests with more input samples or tuning with more trials to show better accuracy and performance benchmark. For nightly, running the test could be triggered based on a timer and implemented however the HW vendor desires. This way Device Test Maintainers have flexibility on the implementation and are not required to make a connection to TVM Jenkins node. + +3. **Test results.** We expect Device Test Maintainers to publicly report functional test results for any on-device tests which also run on simulators in the TVM CI. To facilitate this, TVM will provide reporting infrastructure (i.e. a test dashboard) to present those results in public domain. Our proposal is that Device Test Maintainers upload the tests results in the form of pytest artifacts to an S3 bucket which is provided by TVM community. Device Test Maintainers are also welcome to show the results in the form of a website, but the tests artifacts should be uploaded to the S3 bucket so they can be retrieved in future. + - Other alternative is to use a Github repository to host the test results. Github repo is not the ideal solution for saving and downloading files and it could be slow for hosting large number of files for a website. + +## What is Tested? +Nightly tests could vary based on the target. Some hardware targets have minimal testing in TVM which runs on simulator. For these hardware, Device Test Maintainers should at least run the same tests on physical hardware to validate the simulator tests. In addition, the HW vendor could add any other unit test or full model end-to-end testing which is in the interest of the maintainer or TVM community. In addition they can run existing tests in TVM with modification. For instance, in tuning tests we only run limited trials in the TVM CI, or for accuracy check we only check for limited number of samples. However, nightly regression could run for larger samples or trials to show better accuracy/performance results. + +# Test Tiers +So far we explained a minimal setup to bring a on-device testing CI to TVM on a nightly basis. However, in principle one could enable more frequent testing. TVM defines these tiers: +1. **Tier 1: Run CI for all PRs.** This tier is equivalent to testing support for existing hardware targets that exist in cloud. This case requires large resources to avoid increasing the CI time. TVM community expects close CI infrastructure monitoring if they a Device Test Maintainer registers at this tier. If failures are observed in a CI at this tier which are due to to failures in the CI infrastructure, TVM community expect it to be resolved in one day time frame. If this requirement is not fulfilled the mentioned CI would be degraded to lower tiers. Review Comment: @mehrdadh ah, I see the confusion here. So, I meant two fixes: 1) TVM community expects close CI infrastructure monitoring if they a Device Test Maintainer registers at this tier => TVM community expects close CI infrastructure monitoring if there is a Device Test Maintainer registered at this tier" 2) TVM community expect it to be resolved in one day time frame. => TVM community expects it to be resolved in one-day time frame. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
