gromero commented on code in PR #98: URL: https://github.com/apache/tvm-rfcs/pull/98#discussion_r1087004467
########## rfcs/0098_on_device_testing.md: ########## @@ -0,0 +1,97 @@ +- Feature Name: On-Device Testing in TVM CI +- Start Date: 2023-01-24 +- RFC PR: [apache/tvm-rfcs#0098](https://github.com/apache/tvm-rfcs/pull/0098) +- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000) +- Authors: [Mehrdad Hessar](https://github.com/mehrdadh), [David Riazati](https://github.com/driazati) +# Summary +[summary]: #summary + +This RFC describes the approach and challenges to add non-merge-blocking hardware testing in TVM CI. + +# Motivation +[motivation]: #motivation + +Testing is a major part of any open source project to show its stability to the users and companies who are adopting the project. More than 700 contributors are involved with TVM who works at various companies with different needs/interests in TVM. This means the demand for thorough testing is increasing every day. At the time of writing, TVM tests generally run on the hardware targets when that hardware is available in the cloud (for example, x86 CPU, i386, GPU and AArch64). In addition, TVM has supports hardware targets that are not available in the cloud, such as embedded devices supported by microTVM or the Hexagon DSP. The TVM CI cannot currently test code on those hardware as part of its CI, leaving a gap in testing. + +It is possible for TVM to include on-device tests for these non-cloud devices in its CI. However, because they are not widely available to use in cloud services, blocking PR merges over failures in those tests could impose an undue burden on contributors who don’t have access to that hardware. In that hypothetical world, all contributors would, at some point, need to find a way to debug those tests on such non-cloud hardware, even if they didn’t have access to it. + +This does not mean TVM community cannot still run tests on these hardware, either as part of CI in a non-merge-blocking way or against `main` at an e.g. nightly or post-merge cadence. This RFC aims to find a way for TVM community members with access to those special hardware to be able to expand coverage of TVM CI in an advisory capacity by adding instances of their hardware to TVM CI. + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +We explain this section by following the simplest case of a hardware-in-loop CI testing which is nightly regression testing. Anyone in the community can run nightly regression tests and provide the results to the community. We expect hardware vendors to be one of the parties primarily interested in having nightly testing on their hardware targets that are supported in TVM, but this document refers to anyone running a test as Device Test Maintainers. + +## Test Procedure +There is a minimal set of requirements that TVM community expects Device Test Maintainers to follow. To add nightly tests to TVM, Device Test Maintainers should implement automation that performs the following steps: +1. **Lookup nightly SHA for testing.** To ensure that results from disparate nightly test suites can be compares, an automated nightly process chooses a TVM sha1 which everyone should use. The bot will merge the new daily commits on main branch to TVM `nightly` branch ([PR13564](https://github.com/apache/tvm/pull/13564) implemented this) at 9:00PM PST. Device Test Maintainers should use the sha1 from this `nightly` branch for testing so we have consistent results across multiple CIs. Review Comment: nit: tests suites can be compared (s/compares/compared/) Should we said that the bot will actually sync up (it's push force, not a merge) the `main` branch daily, at 9:00PM PST, to the TVM `nightly`, instead of saying the bot will "merge"? Just for sake of accuracy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
